Equipment control system, equipment control method, and program

Abstract

To provide an equipment control system capable of performing equipment control using a radio wave sensor.SOLUTION: An equipment control system 100 includes: a conversion unit 11 that converts output signals from a radio wave sensor 1 into a piece of location-identifiable information; and a holding unit 12 that holds the location-identifiable information. A difference acquisition unit 13 acquires differences among the multiple pieces of location-identifiable information held. A moving object estimation unit 14 is configured to estimate whether one or more objects are moving or stationary based on the differences, identify the location of an object that is estimated as a moving object, and acquire a piece of location information of the moving object. A dot placement unit 222 places dots corresponding to the location information of the moving object in a virtual space. An equipment control unit controls equipment 200 based on at least dot clouds placed in the virtual space.SELECTED DRAWING: Figure 16

Description

Technical Field

[0001] The present disclosure relates to a device control system, a device control method, and a program. More specifically, the present disclosure relates to a device control system, a device control method, and a program that estimate a moving object (moving body) such as a human body using a radio wave sensor and control a device based on the estimation result.

Background Art

[0002] Patent Document 1 describes an FMCW radar device that transmits a transmission signal frequency-modulated so that the frequency linearly changes with time, receives a reception signal reflected by an object from the transmission signal, and measures the distance to the object from the frequency of a beat signal obtained by multiplying the reception signal and a locally frequency-modulated signal.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] It is conceivable to perform estimation regarding a moving body (for example, estimation as to whether an object is a moving body or a stationary object that does not move, estimation as to whether a moving body is a human body or a moving object other than a human body, identification of the position of a moving body (particularly a human body), and further estimation of the posture of a human body, etc.) using a radio wave sensor that uses radio waves modulated by an FMCW method or the like, and perform device control based on the estimation result. However, such a system or the like has not been conventionally provided.

[0005] An object of the present disclosure is to provide a device control system, a device control method, and a program capable of realizing device control using a radio wave sensor.

Means for Solving the Problems

[0006] A device control system according to one aspect of the present disclosure comprises a radio wave sensor, a conversion unit, a holding unit, a difference acquisition unit, a motion estimation unit, a point placement unit, and a device control unit. The radio wave sensor performs a transmit and receive operation, transmitting a transmission wave, which is a radio wave modulated in a predetermined manner, toward real space, receiving a reflected wave from the real space, and outputting an output signal based on the transmission wave and the reflected wave. The conversion unit performs a conversion process to convert the output signal output by the radio wave sensor into location-identifiable information that can identify the position of one or more objects present in the real space. The holding unit performs a holding process to hold the location-identifiable information converted by the conversion process. The difference acquisition unit performs a difference acquisition process to acquire the difference between a plurality of location-identifiable pieces of information held by the holding process. The motion estimation unit performs a motion estimation process to estimate whether each of the one or more objects is a moving object or a stationary object based on the difference acquired by the difference acquisition process, to identify the position of the object estimated to be a moving object, and to acquire motion position information indicating the identified position. The point placement unit performs a point placement process in which it virtually places points corresponding to the motion position information acquired by the motion estimation process into a virtual space corresponding to the real space. The equipment control unit performs equipment control processing to control the equipment based at least on the point cloud, which is a collection of points placed in the virtual space. The moving body is either a human body or a moving object other than a human body. The moving body estimation process further estimates whether each of the two or more moving body position information obtained with different time differences corresponds to the human body or a moving object other than a human body, based at least on the change in the difference. The point placement process virtually places points in the virtual space that correspond to the moving body position information that the moving body estimation process has estimated to correspond to the human body, from among the two or more moving body position information obtained by the moving body estimation process. A device control system according to one aspect of the present disclosure comprises a radio wave sensor, a conversion unit, a holding unit, a difference acquisition unit, a motion estimation unit, a point placement unit, and a device control unit. The radio wave sensor performs a transmit and receive operation, transmitting a transmission wave, which is a radio wave modulated in a predetermined manner, toward real space, receiving a reflected wave from the real space, and outputting an output signal based on the transmission wave and the reflected wave. The conversion unit performs a conversion process to convert the output signal output by the radio wave sensor into location-identifiable information that can identify the position of one or more objects present in the real space. The holding unit performs a holding process to hold the location-identifiable information converted by the conversion process. The difference acquisition unit performs a difference acquisition process to acquire the difference between a plurality of location-identifiable pieces of information held by the holding process. The motion estimation unit performs a motion estimation process to estimate whether each of the one or more objects is a moving object or a stationary object based on the difference acquired by the difference acquisition process, to identify the position of the object estimated to be a moving object, and to acquire motion position information indicating the identified position. The point placement unit performs a point placement process in which it virtually places points corresponding to the motion position information acquired by the motion estimation process in a virtual space corresponding to the real space. The equipment control unit performs equipment control processing to control the equipment based on at least a point cloud, which is a collection of points placed in the virtual space. The difference acquisition process acquires two or more differences with different time differences between three or more position-identifiable pieces of information at different time points held by the retention process. The motion estimation process acquires two or more pieces of motion position information corresponding to the two or more differences acquired by the difference acquisition process. The point placement process virtually places two or more points corresponding to the two or more pieces of motion position information with different time differences acquired by the motion estimation process in the virtual space.

[0007] A device control method according to one aspect of the present disclosure is a device control method that uses a radio wave sensor to make an estimation regarding a moving object and controls a device based on the estimation result. The radio wave sensor performs a transmit and receive operation in which it transmits a transmission wave, which is a radio wave modulated in a predetermined manner, toward real space, receives a reflected wave from the real space, and outputs an output signal based on the transmission wave and the reflected wave. The device control method comprises a conversion step, a holding step, a difference acquisition step, a moving object estimation step, a point placement step, and a device control step. The conversion step performs a conversion process to convert the output signal output by the radio wave sensor into location-identifiable information that can identify the position of one or more objects present in the real space. The holding step performs a holding process to hold the location-identifiable information converted by the conversion process. The difference acquisition step performs a difference acquisition process to acquire the difference between a plurality of location-identifiable information held by the holding process. The motion estimation step performs a motion estimation process to estimate whether each of the one or more objects is a moving object or a stationary object based on the difference acquired by the difference acquisition process, to identify the position of the object estimated to be a moving object, and to acquire motion position information indicating the identified position. The point placement step performs a point placement process to virtually place points corresponding to the motion position information acquired by the motion estimation process in a virtual space corresponding to the real space. The equipment control step controls the equipment based at least on a point cloud which is a collection of points placed in the virtual space. The moving body is either a human body or a moving object other than a human body. The moving body estimation process further estimates whether each of the two or more moving body position information obtained with different time differences corresponds to the human body or a moving object other than a human body, based at least on the change in the difference. The point placement process virtually places points in the virtual space that correspond to the moving body position information that the moving body estimation process has estimated to correspond to the human body, from among the two or more moving body position information obtained by the moving body estimation process. A device control method according to one aspect of the present disclosure is a device control method that uses a radio wave sensor to make an estimation regarding a moving object and controls a device based on the estimation result. The radio wave sensor performs a transmit and receive operation in which it transmits a transmission wave, which is a radio wave modulated in a predetermined manner, toward real space, receives a reflected wave from the real space, and outputs an output signal based on the transmission wave and the reflected wave. The device control method comprises a conversion step, a holding step, a difference acquisition step, a moving object estimation step, a point placement step, and a device control step. The conversion step performs a conversion process to convert the output signal output by the radio wave sensor into location-identifiable information that can identify the position of one or more objects present in the real space. The holding step performs a holding process to hold the location-identifiable information converted by the conversion process. The difference acquisition step performs a difference acquisition process to acquire the difference between a plurality of location-identifiable information held by the holding process. The motion estimation step performs motion estimation based on the difference acquired by the difference acquisition process to estimate whether each of the one or more objects is a moving object or a stationary object, to identify the position of the object estimated to be a moving object, and to acquire motion position information indicating the identified position. The point placement step performs point placement processing to virtually place points corresponding to the motion position information acquired by the motion estimation process in a virtual space corresponding to the real space. The equipment control step controls the equipment based on at least a point cloud, which is a set of points placed in the virtual space. The difference acquisition process acquires two or more differences with different time differences between three or more position-identifiable pieces of information at different time points held by the retention process. The motion estimation process acquires two or more pieces of motion position information corresponding to the two or more differences acquired by the difference acquisition process. The point placement process virtually places two or more points corresponding to the two or more pieces of motion position information with different time differences acquired by the motion estimation process in the virtual space.

[0008] A program according to one aspect of this disclosure causes one or more processors to execute the device control method. [Effects of the Invention]

[0009] The equipment control system, equipment control method, and program disclosed herein have the effect of enabling equipment control using radio wave sensors. [Brief explanation of the drawing]

[0010] [Figure 1] Figure 1 is a block diagram of an equipment control system (human body posture estimation system) according to an embodiment of this disclosure. [Figure 2] Figure 2 is a conceptual diagram of a room in which the above-mentioned equipment control system is used. [Figure 3] Figure 3 is a graph showing the frequency change of the transmitted wave sent by the radio wave sensor that constitutes the above-mentioned equipment control system. [Figure 4] Figure 4 is a flowchart illustrating the operation of the control device that constitutes the above-mentioned equipment control system. [Figure 5] Figure 5 is a flowchart illustrating the one-to-many frame difference acquisition process included in the above operation. [Figure 6] Figure 6 is a flowchart illustrating the posture estimation process included in the above operation. [Figure 7] Figure 7 is a waveform diagram illustrating an example of inter-frame difference (one-to-one inter-frame difference). [Figure 8] Figure 8A is a frequency spectrum diagram showing the FFT result for the current frame Fr0, Figure 8B is a frequency spectrum diagram showing the FFT result for the subsequent frame Fr1, and Figure 8C is a frequency spectrum diagram showing the difference between the FFT results. [Figure 9] Figure 9 is a waveform diagram illustrating another example of inter-frame differences (one-to-many-frame differences). [Figure 10] Figure 10 is a conceptual diagram illustrating the characteristics of various human body movements. [Figure 11] Figure 11 is a waveform diagram illustrating an example of one-to-many frame difference acquisition corresponding to various movements as described above. [Figure 12] Figure 12A is a distribution map showing an example of cluster distribution in the standing position, one of the postures of the human body; Figure 12B is a distribution map showing an example of cluster distribution in the sitting position; and Figure 12C is a distribution map showing an example of cluster distribution in the lying position. [Figure 13]FIG. 13A is a conceptual diagram showing a three-dimensional figure surrounding a cluster group in a standing position, FIG. 12B is a conceptual diagram showing a three-dimensional figure surrounding a cluster group in a sitting position, and FIG. 12C is a conceptual diagram showing a three-dimensional figure surrounding a cluster group in a lying position. [Figure 14] FIG. 14 is a data structure diagram of an automatic control information group. [Figure 15] FIG. 15 is a data structure diagram of control history information. [Figure 16] FIG. 16 is a block diagram of a modified example of the above-described device control system.

Embodiments for Carrying Out the Invention

[0011] (1) Outline of the Device Control System In this embodiment, the moving body estimation system of the present disclosure is a device control system 100 that also has a device control function in addition to a moving body estimation function (for example, a human body estimation function, particularly a human body posture estimation function). As shown in FIG. 2, the device control system 100 of this embodiment performs estimation regarding a moving object (hereinafter sometimes referred to as a "moving body"), particularly a human body 301, by means of a radio wave sensor 1.

[0012] Estimation regarding the human body 301 includes, for example, estimation as to whether it is the human body 301, a moving object other than the human body 301, a non-moving object (stationary object), etc., and further includes estimation of the posture of the human body 301.

[0013] The posture of the human body 301 is, for example, a standing position, a sitting position, a lying position (see FIGS. 12A to 12C), etc., but is not limited thereto. Then, based on the estimated change in posture, the device control system 100 detects various movements of the human body 301, and thus the behavior of a person, and controls the device 200 according to the detection result.

[0014] Postural changes include, but are not limited to, changes between standing, sitting, and lying positions. Postural changes also include the continuation of no change in posture (for example, when a predetermined period of time has passed since changing to a sitting position, and no change back to standing, etc., occurs). Furthermore, the behaviors to be detected are, in particular, non-manipulative behaviors (described later), but manipulative behaviors (described later) may also be used.

[0015] (1-1) Radio wave sensor The radio wave sensor 1 transmits radio waves (transmitted wave Tr) from its antenna, receives the reflected wave Re, which is the transmitted wave Tr reflected by the object, with its antenna, and outputs information that can identify the location of the object (hereinafter referred to as "location-identifiable information").

[0016] (1-1-1) Location-identifiable information Location-identifiable information refers to information that allows the location of an object to be identified. The object's location is preferably in three dimensions, but may also be in two or one dimensions. Location-identifiable information may be, for example, the FFT result set described later (information that allows the location of a three-dimensional object), but it may also be each of the multiple (e.g., three) FFT results constituting the FFT result set (information that allows the location of a one-dimensional object).

[0017] Furthermore, a signal indicating the time difference Δt from the transmission of the transmitting wave Tr to the reception of the reflected wave Re, or a signal indicating the frequency difference Δf corresponding to the time difference Δt (for example, an IF signal indicating the frequency difference Δf between the transmitting wave Tr modulated in the FMCW method and the reflected wave Re: see Figure 3), can also be considered a type of location-determining information. Moreover, the distance calculated from the time difference Δt or frequency difference Δf, etc., may itself be location-determining information.

[0018] (1-1-2) Antenna Furthermore, the antenna used for transmitting the transmitted wave Tr and the antenna used for receiving the reflected wave Re may be the same antenna (hereinafter referred to as the "shared antenna") or separate antennas (hereinafter referred to as the "transmitting antenna" and the "receiving antenna"). In other words, the transmitted wave Tr may be transmitted from the shared antenna and the reflected wave Re corresponding to the transmitted wave Tr may be received by the shared antenna, or the transmitted wave Tr may be transmitted from the transmitting antenna and the reflected wave Re corresponding to the transmitted wave Tr may be received by the receiving antenna.

[0019] Generally, in order to determine the three-dimensional position of an object (for example, to place a point corresponding to the human body in a virtual three-dimensional space 500), the radio wave sensor 1 needs to have, for example, three or more shared antennas, or one transmitting antenna and three or more receiving antennas. Note that the presence of multiple antennas in the radio wave sensor 1 is not limited to cases where the multiple antennas are housed in a single enclosure, but also includes cases where they are arranged in multiple locations that are far apart from each other. In the latter case, each of the multiple antennas is connected to the sensor body via wired or wireless means to enable communication.

[0020] The radio wave sensor 1 in this embodiment has one transmitting antenna and three or more (for example, three) receiving antennas. The position of each of the one transmitting antenna and the three or more receiving antennas included in the radio wave sensor 1 is known, and four or more pieces of positional information corresponding to these four or more antennas (hereinafter referred to as the "antenna positional information group") are stored in advance in the memory of the control device 2, for example.

[0021] However, the number of receiving antennas constituting the radio wave sensor 1 may be two or one (if there are two, they can be arranged as points on a plane; if there is one, they can be arranged as points along a line).

[0022] (1-1-3) Sending and receiving operation The radio wave sensor 1 performs a transmit and receive operation. The transmit and receive operation involves transmitting a transmission wave Tr toward the real space where the group of objects may exist, receiving a reflected wave Re from the real space, and outputting a signal (e.g., an IF signal) based on the transmitted wave Tr and the reflected wave Re. The real space where the group of objects may exist is, for example, the space enclosed by the floor, ceiling, and side walls of room 400 (indoor space), and may be referred to as "real space (400)" below. Note that the real space where the group of objects may exist is not limited to indoor space, but may also be an outdoor space such as a corridor or terrace.

[0023] A group of objects is a collection of one or more objects. In this embodiment, the group of objects includes one or more of the following: a human body 301, a moving object other than a human body (for example, an electric blind 200c; hereafter, this may be referred to as "a moving object other than a human body (200c)"), and a stationary object 302. In other words, each of the one or more objects constituting the group of objects is either a human body 301, a moving object other than a human body (200c), or a stationary object 302.

[0024] The radio wave sensor 1 performs, for example, the following transmission and reception operation at a predetermined interval (for example, once every 20ms). This transmission and reception operation involves transmitting a transmission wave Tr to the real space (400) where the group of objects may exist using one transmitting antenna, and receiving the reflected wave Re from the real space (400) with each of three or more receiving antennas.

[0025] For example, if a human body 301, a moving object other than a human body (200c), and a stationary object 302 exist in real space (400), reflected waves Re from the group of objects (301, 200c, 302) are received by each of the three or more receiving antennas. The stationary object 302 is, for example, a desk 302a and a bed 302b placed on the floor of room 400 in the example in Figure 2, but it may also be the floor itself.

[0026] The transmitting wave Tr is a radio wave modulated using a predetermined method. The predetermined method is, for example, the FMCW (Frequency Modulated Continuous Wave) method, but is not limited to this. In this embodiment, the transmitting wave Tr is a radio wave modulated using the FMCW method.

[0027] (1-1-4)FMCW method The FMCW method, as shown in Figure 3 for example, is a method in which the frequency f of a transmitted wave Tr (transmitted signal) having a predetermined time length (chirp length Tc of the chirp signal) is linearly increased (or decreased) from the starting frequency f0 with a predetermined slope S as time t progresses.

[0028] Furthermore, the acquisition of location-identifiable information as described above does not have to be performed for all transmission and reception operations that are repeated at a predetermined interval. For example, as will be described later, if a predetermined time T (for example, T = 200 ms) is defined as one frame (described later), and N transmission and reception operations (N is a natural number: for example, 10 times) are performed in one frame, location-identifiable information may be acquired for each of the N transmission and reception operations belonging to one frame, or it may be acquired for only one predetermined operation (for example, the first transmission and reception operation).

[0029] The predetermined period is, for example, once every 20ms. In this embodiment, with one frame being 200ms, this corresponds to a frequency of 10 times per frame (N=10). However, the predetermined period may also be, for example, 20 times per frame (N=20), 5 times per frame (N=5), etc.

[0030] (1-1-5) IF signal, FFT result, and FFT result group Each time the radio wave sensor 1 performs a transmit / receive operation, it generates an IF signal for each of the three or more receiving antennas, applies an FFT (Fast Fourier Transform) to the IF signal to obtain the FFT result, and outputs a set of FFT results. In other words, each time a transmit / receive operation is performed, the radio wave sensor 1 outputs a set of FFT results consisting of three or more FFT results corresponding to the three or more receiving antennas. However, a Fourier transform other than FFT may be applied to the IF signal, in which case the radio wave sensor 1 outputs a set of Fourier transform results consisting of three or more Fourier transform results corresponding to the three or more receiving antennas.

[0031] The IF signal, as shown in Figure 3, is a signal that indicates the frequency difference Δf between the transmitted wave Tr and the reflected wave Re. The IF signal is a signal that indicates the difference between the frequency t of the transmitted wave Tr and the frequency t of the reflected wave Re at time t, and is a function of time t Δf(t), but when the group of objects (301,302) is stationary, it shows a constant value.

[0032] The IF signal is generated by mixing the transmitted wave Tr and the reflected wave Re. The IF signal is generated during the period when the transmitted wave Tr is being transmitted and the reflected wave Re is being received (i.e., from the start of receiving the reflected wave Re to the end of transmitting the transmitted wave Tr).

[0033] The FFT result is the result of applying an FFT to an IF signal. The FFT result provides information such as the frequency spectrum (relationship between frequency f and reflection intensity amp), as shown in Figures 8A and 8B.

[0034] An FFT result set is information consisting of three or more FFT results corresponding to three or more receiving antennas, acquired for a single transmission and reception operation. Such an FFT result set makes it possible to determine the three-dimensional position of an object. Furthermore, by taking the difference between multiple FFT result sets, frequency components corresponding to stationary objects 302 are removed, and only frequency components corresponding to moving objects such as a human body 301 (frequency f where amp exceeds a threshold, and the value of amp corresponding to that frequency f) are acquired (see Figures 8A to 8C). Based on these acquired frequency components, it becomes possible to obtain information regarding the three-dimensional position and movement of moving objects such as a human body 301.

[0035] In this embodiment, the radio wave sensor 1 outputs multiple FFT result groups corresponding to a series of multiple transmission and reception operations, and these output FFT result groups are stored in time series in the memory of the control device 2. Meanwhile, the memory also stores the antenna position information group described above. The control device 2 then calculates the time difference (time difference of the FFT result for each of the three receiving antennas) between the multiple FFT result groups (for example, two adjacent FFT result groups) stored in time series in the memory, and obtains a group of distance measurement results (two distance measurement results corresponding to the three receiving antennas) based on the calculated difference. The control device 2 then performs three-point positioning using the distance measurement result group thus obtained and the pre-stored antenna position information group. This makes it possible to obtain information to identify the three-dimensional position of the human body 301, for example, to calculate three-dimensional coordinates.

[0036] (1-2) Attitude estimation function of the equipment control system As shown in Figure 1, the equipment control system 100 includes a distance measuring unit 221, a point placement unit 222, and an attitude estimation unit 223.

[0037] (1-2-1) Distance measuring unit: one-to-one or one-to-many difference The distance measuring unit 221 measures the distance from the radio wave sensor 1 to the human body 301 based on the FFT results output by the radio wave sensor 1.

[0038] More specifically, the distance measuring unit 221 holds the FFT result group output by the radio wave sensor 1 for a period of time longer than the predetermined period (preferably, a period of time twice the predetermined period or longer). The distance measuring unit 221 then performs distance measurement processing based on the difference between one of the FFT result groups held, which is the reference FFT result group (in this embodiment, the current FFT result group), and at least one target FFT result group (in this embodiment, at least one preceding FFT result group), and obtains at least one distance measurement result group.

[0039] (1-2-1a) Standard FFT result group and target FFT result group The reference FFT result group is one of several FFT result groups held by the distance measuring unit 221 that serves as the starting point for acquiring the difference. The target FFT result group is each of the one or more FFT result groups held by the distance measuring unit 221 that are the targets for acquiring the difference between them and the reference FFT result group. The one or more target FFT groups are located either before or after (usually before) the reference FFT group in time.

[0040] In this embodiment, one FFT result group is the current FFT result group (described later), and one or more target FFT result groups are one or more prior FFT result groups (described later).

[0041] By taking the difference between the reference FFT result set and at least one target FFT result set, the reflection component from the stationary object 302 is removed from the reflected wave Re (the reflected intensity, which is the received intensity of the reflected wave Re, becomes below the threshold), and only the reflection component from the human body 301 (the moving object) remains (at least one set, preferably two or more sets, of a reflected intensity amp exceeding the threshold and a frequency f corresponding to that reflected intensity are detected).

[0042] Furthermore, the number of preceding FFT result groups, i.e., the number of preceding FFT result groups from which the difference between the current FFT result group and "at least one" is obtained, is preferably two or more (one-to-many inter-frame differences: one-to-five inter-frame differences in the illustrated examples), as shown in Figures 9 and 11, in terms of improving resolution, but it may also be just one. In other words, even when taking a one-to-one difference (one-to-one inter-frame difference) as shown in Figures 7 and 8A to 8C, attitude estimation is possible, and resolution can also be improved by increasing the number of receiving antennas (hereinafter, "number of antennas").

[0043] (1-2-1b) Current FFT result group and previous FFT result group The current FFT result set refers to the most recent FFT result set among the multiple FFT result sets that are held. The preceding FFT result set refers to the FFT result sets that are held before the current FFT result set. By designating the reference FFT result set as the current FFT result set and the target FFT result set as the preceding FFT result set, real-time pose estimation can be achieved.

[0044] (1-2-1c) Distance measurement process Distance measurement processing is a process that uses the radio wave sensor 1 to measure the distance to an object such as a human body 301. In the distance measurement processing in this embodiment, the distance from each of the three or more receiving antennas constituting the radio wave sensor 1 (hereinafter, each antenna) to the human body 301 is measured, and three or more distance measurement results (group of distance measurement results: described later) corresponding to the three or more receiving antennas are obtained.

[0045] In the distance measurement process, for example, the distance d from each antenna to the human body 301 is calculated by the following equation 1, based on the various parameters shown in Figure 3, namely the difference (frequency difference Δf) between the transmitted wave Tr and the reflected wave Re, the slope S in the linear change of the frequency f of the transmitted wave Tr, and the speed of light c. d=(c / 2S)×Δf...(Formula 1)

[0046] (1-2-1d) Range measurement results group The distance measurement result group is information consisting of three or more distance measurement results corresponding to three or more receiving antennas, acquired using Equation 1 above for a single transmission and reception operation.

[0047] Three or more distance measurement results corresponding to three or more receiving antennas refer, for example, to three distance measurement results, numbered 1 to 3, corresponding to three receiving antennas, numbered 1 to 3. The first distance measurement result corresponds to approximately half the propagation distance of radio waves radiated from the transmitting antenna, reflected by the human body 301, and reaching the first receiving antenna (i.e., the distance from the human body 301 to the first receiving antenna). The second distance measurement result corresponds to approximately half the propagation distance of radio waves radiated from the transmitting antenna, reflected by the human body 301, and reaching the second receiving antenna (i.e., the distance from the human body 301 to the second receiving antenna). The third distance measurement result corresponds to approximately half the propagation distance of radio waves radiated from the transmitting antenna, reflected by the human body 301, and reaching the third receiving antenna (i.e., the distance from the human body 301 to the third receiving antenna).

[0048] Furthermore, the number of distance measurement result groups acquired, i.e., "at least one" group of distance measurement results, is preferably two or more, but may be just one.

[0049] (1-2-2) Point arrangement section: Multi-point arrangement The point placement unit 222 performs coordinate calculation processing each time the distance measurement unit 221 performs distance measurement processing, and places at least one point corresponding to the current FFT result group in the three-dimensional space 500, for example, as shown in Figure 12A. However, the placement location of the points may also be in two-dimensional space (plane) or one-dimensional space (line).

[0050] Furthermore, the number of points that are "at least one," that is, the number of points that are placed at once corresponding to the current FFT result set, is preferably two or more (multiple point placement), but it may also be one (single point placement).

[0051] (1-2-2a) Three-dimensional space The three-dimensional space 500 is a virtual space corresponding to the real space (for example, room 400: see Figure 2) in which the radio wave sensor 1 and the group of objects (301, 302) exist. Placement in the three-dimensional space 500 can be done through virtual actions or simply by storing the three-dimensional coordinates.

[0052] (1-2-2b) Coordinate calculation process and point cloud The coordinate calculation process is the process of calculating the three-dimensional coordinates of the human body 301 based on at least one set of distance measurement results obtained.

[0053] (1-2-3) Posture estimation section The posture estimation unit 223 estimates the posture of the human body 301 (in this embodiment, whether the posture is standing, sitting, or lying down) based on a point cloud 501, for example, as shown in Figures 12A to 12C. The point cloud 501 is a set of one or more points arranged in three-dimensional space 500 by the point placement unit 222.

[0054] Thus, according to this embodiment, it is possible to estimate the posture of a human body 301 using a radio wave sensor 1 based on the FMCW method.

[0055] (2) Details of the distance measuring unit and point placement unit (2-1) One-to-many difference Preferably, the distance measuring unit 221 holds the FFT result group output by the radio wave sensor 1 for a period of at least twice the predetermined period. The period of at least twice the predetermined period may be, for example, at least twice the length of one frame (2 × T).

[0056] The distance measuring unit 221 then calculates the difference between the current FFT result group and each of the two or more preceding FFT result groups from the three or more FFT result groups it currently holds, and performs distance measuring processing based on each of the two or more calculated differences to obtain two or more distance measuring result groups.

[0057] The point placement unit 222 performs coordinate calculation processing based on each of the two or more distance measurement result groups acquired by the distance measurement unit 221, thereby placing two or more points corresponding to the current FFT result group in the three-dimensional space 500.

[0058] In this way, by calculating the difference between the current FFT result set and each of two or more preceding FFT result sets, it is possible to improve the accuracy of estimating the posture of the human body 301 without increasing the number of antennas of the radio wave sensor 1.

[0059] (2-2) One-to-many difference suitable for detecting various human body movements The distance measuring unit 221 more preferably holds the FFT result group output by the radio wave sensor 1 for a period of three times or more times a predetermined period. The distance measuring unit 221 then calculates the difference between the current FFT result group and each of the three or more preceding FFT result groups from the four or more FFT result groups it currently holds, and selects two or more differences from the three or more calculated differences that correspond to various movements of the human body 301 (e.g., body movement, respiratory tremors, limb movements, etc.). The distance measuring unit 221 obtains two or more distance measurement result groups by performing the distance measurement process based on each of the two or more differences thus selected.

[0060] The point placement unit 222 places two or more points corresponding to the current FFT result group in the three-dimensional space 500 by performing the coordinate calculation process based on each of the two or more distance measurement result groups acquired by the distance measurement unit 221.

[0061] In this way, by calculating the difference between the current FFT result and each of three or more preceding FFT results, and selecting two or more differences corresponding to various movements of the human body 301, it is possible to improve the accuracy of estimating the posture of the human body 301 without increasing the number of antennas and while suppressing the number of differences used in the distance measurement process.

[0062] (2-3) Frames and inter-frame differences The aforementioned transmission and reception operations are performed N times per frame (where N is an integer greater than or equal to 2), with a predetermined time frame representing one frame. In this embodiment, the predetermined time is 200ms and N=10.

[0063] The memory stores N sets of FFT results per frame, spanning multiple frames. Specifically, the radio wave sensor 1 performs, for example, 10 transmission and reception operations per frame, and the control device 2's memory stores 10 sets of FFT results per frame, spanning multiple frames (for example, 6 frames). The distance measuring unit 221 acquires the inter-frame difference between the multiple frames stored in the memory.

[0064] The inter-frame difference is the difference between the FFT results belonging to one frame (for example, the reference frame Fr0 shown in Figure 7) and the FFT results belonging to one or more other frames (for example, the subsequent frames Fr1... shown in Figure 7).

[0065] (2-3-1) One-to-one frame difference Inter-frame differences are, for example, one-to-one inter-frame differences. A one-to-one inter-frame difference is the difference between the set of FFT results belonging to one frame (e.g., reference frame Fr0) and the set of FFT results belonging to another frame (e.g., subsequent frame Fr1).

[0066] (2-3-2) Representative values ​​for each frame when calculating the difference between frames Each of the two FFT result sets corresponding to the two frames from which the difference is to be obtained (for example, the FFT result set belonging to the reference frame Fr0 and the FFT result set belonging to the subsequent frame Fr1) is a representative value among the N (for example, 10) FFT result sets in the frame to which it belongs.

[0067] The representative value is, for example, the average value of N FFT result groups (specifically, information consisting of the average value of 10 FFT results corresponding to the first receiving antenna, the average value of 10 FFT results corresponding to the second receiving antenna, and the average value of 10 FFT results corresponding to the third receiving antenna, etc.). In this case, the inter-frame difference is the difference between the average value of the 10 FFT result groups belonging to the reference frame Fr0 and the average value of the 10 FFT result groups belonging to the subsequent frame Fr1.

[0068] Alternatively, the representative value may be one FFT result group (for example, the k-th FFT result group, where k is an integer between 1 and N) selected by a predetermined rule from among N FFT result groups. In this case, the inter-frame difference is the difference between the k-th (for example, the 1st) FFT result group from the 10 FFT result groups belonging to the reference frame Fr0 and the k-th (for example, the 1st) FFT result group from the 10 FFT result groups belonging to the subsequent frame Fr1.

[0069] Furthermore, the above-mentioned matters concerning inter-frame differences apply not only to one-to-one inter-frame differences but also to one-to-many inter-frame differences.

[0070] The distance measuring unit 221, for example, in order to calculate the one-to-many frame difference (described later), holds the FFT result group output by the radio wave sensor 1 for a period of (K+1) frames (where K is an integer of 2 or more). In this embodiment, K=5, and the FFT result group is held for a period of (5+1) frames, i.e., 6 × 200 ms = 1200 ms (for example, stored in the memory of the control device 2).

[0071] (2-3-3) One-to-many frame difference The distance measuring unit 221 calculates the one-to-many frame difference using, for example, a set of FFT results over a (K+1) frame period stored in memory. The one-to-many frame difference is the difference in the set of FFT results (first time difference T, second time difference 2×T, ...) between the reference frame Fr0 and each of the multiple target frames (for example, subsequent frames Fr1, Fr2, ... as shown in Figure 9).

[0072] In Figure 9, multiple target frames are shown as examples of multiple subsequent frames Fr1, Fr2, ... following the reference frame Fr0. However, in this embodiment, multiple target frames are, for example, multiple preceding frames Fr-1, Fr-2, ... preceding the reference frame Fr0, as shown in Figure 11.

[0073] The difference between the FFT result sets is, for example, the difference between the FFT result set corresponding to the first transmitted wave Tr1 of the reference frame Fr0, as shown in Figure 7, and the FFT result set corresponding to the first transmitted wave Tr1 of the target frame with a time difference T relative to the reference frame Fr0 (in the example in Figure 7, the subsequent frame Fr1).

[0074] Alternatively, the difference between the FFT result sets could be, for example, the difference between the FFT result set corresponding to the second transmitted wave Tr2 of the reference frame Fr0 and the FFT result set corresponding to the second transmitted wave Tr2 of the target frame (successor frame Fr1), or it could be the difference between the FFT result set corresponding to the Nth transmitted wave TrN of the reference frame Fr0 and the FFT result set corresponding to the Nth transmitted wave Tr2 of the target frame (successor frame Fr1).

[0075] Alternatively, the difference of the FFT results may be the sum of the N differences as described above. In this embodiment, the difference of the FFT results is assumed to be the sum of the N differences as described above, that is, the sum of the differences for each of the N transmitted waves Tr1 to TrN.

[0076] In this embodiment, the one-to-many frame difference is a set of frame differences between the current frame FFT result group and each of the K or more (for example, 5) preceding frame FFT result groups, from the currently held (N × (K+1) or more) FFT result groups (for example, 10 × (5+1) = 60) FFT result groups, where N=10 and K=5.

[0077] (2-3-4) Current frame and current frame FFT results The current frame Fr0 is the current frame, and the current frame is frame Fr0 which contains the most recent set of FFT results. The current frame FFT results set is the set of N FFT results belonging to the current frame Fr0.

[0078] (2-3-5) Preceding frames and the FFT results of the preceding frames The preceding frames (Fr-1, Fr-2...Fr-K) are frames prior to the current frame Fr0. The set of preceding frame FFT results is the set of N FFT result sets, each belonging to one of the K preceding frames (Fr-1, Fr-2...Fr-K).

[0079] (2-3-6) Parameter K Furthermore, the larger the value of parameter K, the easier it becomes to detect various movements of the human body 301 (e.g., body movement, respiratory tremors, and limb movements).

[0080] (2-3-6a) Body movements, respiratory tremors, and limb movements Body movement refers to the movement of the entire body (trunk). As shown in Figures 10 and 11, body movement is irregular, with long durations and large movement amounts. Limb movement refers to the movement of the arms and legs. As shown in Figures 10 and 11, limb movement is irregular, with short durations and large movement amounts. Respiratory tremors refer to the movement of the torso associated with breathing. Respiratory tremors are periodic, with small movement amounts and relatively short durations.

[0081] (2-3-6b) Specific examples of parameter K In this embodiment, as shown in Figure 11, K=5, and three frames (the second, third, and fifth preceding frames Fr-2, Fr-3, and Fr-5) suitable for detecting body movement, limb movement, and respiratory tremors are selected from five consecutive preceding frames (first preceding frame Fr-1, second preceding frame Fr-2, ... fifth preceding frame Fr-5). This reduces the processing load for difference calculation while improving estimation accuracy. However, all five preceding frames Fr-1 to Fr-5 may also be used.

[0082] (2-3-7) Pose estimation based on one-to-many frame differences The distance measuring unit 221 acquires a group of K or more distance measuring results corresponding to the K or more inter-frame differences that constitute the one-to-many inter-frame differences calculated with respect to the current frame Fr0.

[0083] The point placement unit 222 performs coordinate calculation processing based on each of the K or more distance measurement result groups acquired by the distance measurement unit 221 for the current frame Fr0, thereby placing K or more points corresponding to the current FFT result group in the three-dimensional space 500.

[0084] In this way, by calculating the one-to-many frame difference for each of two or more preceding FFT result sets of the current FFT result set, it is possible to further improve the accuracy of estimating the posture of the human body 301 without increasing the number of antennas.

[0085] (2-3-8) Preferred values ​​for parameter K The value of K is preferably an integer of 3 or greater. The distance measuring unit 221 selects two or more inter-frame differences from the three or more inter-frame differences that constitute the one-to-many inter-frame difference calculated with respect to the current frame Fr0, corresponding to various movements of the human body 301 (e.g., body movement, respiratory tremors, limb movements, etc.), and obtains two or more groups of distance measurement results corresponding to the two or more selected inter-frame differences.

[0086] The point placement unit 222 performs coordinate calculation processing based on each of the two or more distance measurement result groups acquired by the distance measurement unit 221 for the current frame Fr0, thereby placing two or more points corresponding to the current FFT result group in the three-dimensional space 500.

[0087] In this way, by calculating three or more inter-frame differences (one-to-many inter-frame differences) for each of three or more preceding FFT result sets of the current FFT result set, and then calculating two or more inter-frame differences from these three or more inter-frame differences corresponding to various movements of the human body 301, it is possible to further improve the accuracy of estimating the posture of the human body 301 without increasing the number of antennas and while suppressing the number of inter-frame differences used for distance measurement.

[0088] A more preferable value for K is 5. The distance measuring unit 221 selects three inter-frame differences from the five inter-frame differences that constitute the one-to-many inter-frame difference calculated for the current frame Fr0, corresponding to the body movement, respiratory tremors, and limb movements of the human body 301, and acquires three groups of distance measurement results corresponding to the three selected inter-frame differences.

[0089] The three inter-frame differences corresponding to the body movements, respiratory tremors, and limb movements of the human body 301 are, in terms of unity, for example, the inter-frame differences ΔA2, ΔA3, and ΔA5 of the inter-frame differences ΔA1 to ΔA5 between the first to fifth frames, as shown in Figure 11.

[0090] The point placement unit 222 performs coordinate calculation processing based on each of the three distance measurement result groups acquired by the distance measurement unit 221 for the current frame, thereby placing the three points corresponding to the current FFT result group in the three-dimensional space 500.

[0091] In this way, by calculating five inter-frame differences (one-to-many inter-frame differences) for each of the five preceding FFT result sets of the current FFT result set, and then calculating three inter-frame differences from these five differences that correspond to the body movements, respiratory tremors, and limb movements of the human body 301, it is possible to further improve the accuracy of estimating the posture of the human body 301 without increasing the number of antennas and while suppressing the number of inter-frame differences used for distance measurement.

[0092] The above is merely an example; for example, the number of inter-frame differences and the time difference calculated can be modified as appropriate to improve estimation accuracy.

[0093] (2-3-9) Details of the posture estimation unit (2-3-9a) Clustering and distribution of cluster groups The attitude estimation unit 223 obtains a cluster group (CL, CL1, CL2), which is a set of one or more clusters (CL, CL1, CL2), by, for example, performing clustering on the point cloud 501.

[0094] The cluster group (CL, CL1, CL2) is, for example, a first cluster CL1 corresponding to the upper body (head, torso, and arms) and a second cluster CL2 corresponding to the lower body (legs) of the human body 301, as shown in Figures 12A and 12B. Alternatively, the cluster group (CL, CL1, CL2) may be a single cluster CL corresponding to the entire body (head, torso, arms, and legs), as shown in Figure 12C.

[0095] Furthermore, as the number of points constituting the point cloud 501 increases, it is expected that the cluster groups (CL, CL1, CL2) will have a resolution sufficient to distinguish the general shape (silhouette) of the human body 301, and by extension, the individual parts that make up the human body 301.

[0096] The posture estimation unit 223 estimates the posture of the human body 301 based on the distribution of the cluster group (CL, CL1, CL2) acquired in this way within the three-dimensional space 500 (hereinafter simply referred to as "distribution").

[0097] The distribution includes, for example, the number of clusters (CL, CL1, CL2) obtained, the direction of spread of a single cluster (CL, CL1, CL2), and the distance between multiple clusters.

[0098] (2-3-9b) Pose estimation based on distribution conditions The posture estimation unit 223 estimates whether the posture of the human body 301 is supine, sitting, or standing, based on, for example, distribution conditions, which are conditions related to distribution.

[0099] The distribution conditions include, for example, a supine condition. The supine condition (first supine condition) that constitutes the distribution conditions is, for example, "only a single cluster CL is acquired, and the height of that single cluster CL from the floor is low enough to fall below a threshold" (see Figure 12C).

[0100] The distribution conditions may further include, for example, a locus condition. The locus condition that constitutes the distribution conditions (first locus condition) is, for example, "two clusters, CL1 and CL2, are acquired, and the distance between these two clusters, CL1 and CL2, is close enough to fall below a threshold" (see Figure 12B).

[0101] The posture estimation unit 223 may, for example, determine that the posture of the human body 301 is lying down if the distribution satisfies the supine condition (first supine condition), determine that the posture is sitting if the distribution satisfies the sitting condition (first sitting condition), and determine that the posture is standing if the distribution does not satisfy either the supine condition or the sitting condition.

[0102] In this way, the posture of 301 human bodies can be accurately estimated based on the distribution of cluster groups (CL, CL1, CL2).

[0103] (2-3-9c) Pose estimation based on dimensional ratio conditions Alternatively, the posture estimation unit 223 may estimate the posture of the human body 301 based on dimensional ratio conditions relating to the ratios of the length D, width W, and height H of the three-dimensional figure 502 surrounding the cluster group (CL, CL1, CL2) (hereinafter referred to as "dimensional ratio").

[0104] The three-dimensional figure 502 in this embodiment is, for example, a rectangular parallelepiped with length D, width W, and height H, as shown in Figures 13A to 13C, and its dimensional ratio is, for example, the ratio between length D or width W and height H.

[0105] However, the three-dimensional figure 502 could also be, for example, an elliptical cylinder (not shown). In the case of an elliptical cylinder, the minor axis corresponds to the vertical D and the major axis corresponds to the horizontal W.

[0106] The dimensional ratio condition includes the supine position condition. The supine position condition (second supine position condition) that constitutes the dimensional ratio condition may be, for example, "the vertical D or horizontal W of the three-dimensional figure 502 is larger than a predetermined ratio (for example, 3 times) relative to the height H."

[0107] The dimensional ratio condition may further include, for example, an upright condition. The upright condition that constitutes the dimensional ratio condition may be, for example, "the height H of the three-dimensional figure 502 is larger than a predetermined ratio (for example, 3 times) of the length D or width W."

[0108] The posture estimation unit 223 may, for example, determine that the posture of the human body 301 is in a lying position if the dimensional ratio satisfies the above-mentioned lying position condition (second lying position condition), determine that the posture is in an upright position if the dimensional ratio satisfies the above-mentioned standing position condition, and determine that the posture is in a seated position if the dimensional ratio does not satisfy either the above-mentioned lying position condition or the above-mentioned standing position condition.

[0109] In this way, the posture of the human body 301 can be accurately estimated based on the ratios of the length D, width W, and height H of the three-dimensional figure 502 surrounding the cluster group (CL, CL1, CL2).

[0110] (2-3-9d) Pose estimation based on distribution dimension ratio conditions The posture estimation unit 223 may, for example, estimate whether the posture of the human body 301 is supine, sitting, or standing, based on the distribution and dimension ratio conditions, which are conditions relating to the distribution and dimension ratio.

[0111] The distribution dimension ratio condition includes the supine condition. The supine condition (third supine condition) that constitutes the distribution dimension ratio condition may be, for example, "only one cluster CL is obtained, and the vertical D or horizontal W of the three-dimensional figure 502 is larger than a predetermined ratio (for example, 3 times) to the height H."

[0112] The distribution dimensional ratio condition further includes, for example, a seating condition. The seating condition (second seating condition) that constitutes the distribution dimensional ratio condition may be, for example, "two clusters, CL1 and CL2, are acquired, and the distance between these two clusters, CL1 and CL2, is close enough to fall below a threshold."

[0113] The posture estimation unit 223 may, for example, determine that the posture of the human body 301 is lying down if the distribution and dimensional ratio satisfy the supine condition (third supine condition), determine that the posture is sitting if the distribution and dimensional ratio satisfy the sitting condition (second sitting condition), and determine that the posture is standing if the distribution and dimensional ratio do not satisfy either the supine condition or the sitting condition.

[0114] (2-4) Equipment control functions of the equipment control system As shown in Figure 1, the device control system 100 further comprises an action detection unit 224 and a device control unit 225.

[0115] (2-4-1) Behavior detection unit The behavior detection unit 224 detects the actions of a person corresponding to the human body 301 based on the estimation results of the posture estimation unit 223 for the human body 301.

[0116] In this embodiment, the behavior detection unit 224 detects non-operational behavior, in particular, based at least on changes in the estimation results of the posture estimation unit 223.

[0117] In this embodiment, the changes in the estimation results are changes between standing, sitting, and lying positions, specifically, for example, changes from standing to sitting, changes from sitting to standing, changes from standing or sitting to lying down, changes from lying down to standing or sitting, and so on.

[0118] Furthermore, the behavior detection unit 224 may detect non-operational behavior if a state of no change is detected for a predetermined period of time or longer. For example, the behavior detection unit 224 detects non-operational behavior after a change in the estimation result of the posture estimation unit 223 has been detected, and the state of no change has continued for a predetermined period of time or longer.

[0119] Specifically, the behavior detection unit 224 detects a non-operational behavior, such as continuing to lie down (e.g., falling asleep), when, for example, the estimation result of the posture estimation unit 223 changes from standing or sitting to lying down, and a state of no change without detection of any change from lying down to another posture continues for a predetermined period of time or longer. Also, the behavior detection unit 224 detects a non-operational behavior, such as continuing to lie down (e.g., falling asleep), when, for example, the estimation result of the posture estimation unit 223 changes from standing or lying down to sitting, and a state of no change without detection of any change from sitting to another posture continues for a predetermined period of time or longer.

[0120] (2-4-1a) Non-manipulative behavior and manipulative behavior Non-manipulative behavior is a behavior that differs from manipulative behavior.

[0121] An operating action is an action intended to operate the device 200. The device 200 is, for example, a lighting fixture 200a mounted on the ceiling of room 400, a television (TV) 200b placed on the floor, and an electric blind 200c attached to window 402, as shown in Figure 2. Operating actions include, for example, turning the lighting fixture 200a on and off via a wall switch, turning the TV 200b on and off and changing channels via a remote control, and opening and closing the electric blind 200c via a remote control.

[0122] Non-operational actions are actions that do not involve operating the device 200, such as daily activities. Examples of daily activities include entering room 400 (entering through the doorway 401 of room 400), lying down on bed 302b (going to sleep), getting up from bed 302b (getting up), and leaving room 400 (leaving through doorway 401). Other examples of daily activities include sitting at desk 302a (sitting down) (working at the desk or relaxing), and leaving desk 302a (leaving the desk).

[0123] Furthermore, seating may be distinguished into, for example, seating for working at desk 302a (desk work) and seating for relaxing while watching TV 200b (relaxation). Desk work and relaxation may be determined, for example, based on the time of day when seating is detected (daytime or nighttime).

[0124] The behavior detection unit 224 may also detect non-operational behaviors by taking into consideration environmental information, historical information, etc. (described later). For example, waking up may be distinguished into waking up in the morning and waking up in the middle of the night. Whether it is waking up in the morning or waking up in the middle of the night may be determined based on the time period during which a change from lying down to sitting or standing was detected.

[0125] (2-4-2) Equipment Control Unit The device control unit 225 controls the device 200 based at least on the non-operational behavior detected by the behavior detection unit 224.

[0126] In addition to control in response to non-operational actions (automatic control), the device control unit 225 also performs control in response to operations on the device 200 (manual control). Operations may include, for example, operation of an operation device (touch panel, operation button, etc.) on the control device 2, operation of an operation device (wall switch, remote control, etc.) on the device 200, or gestures for operating the device 200. Operations of an operation device on the device 200 are detected, for example, by intercepting control signals from an operation device such as a wall switch to the device 200. Gestures are detected, for example, by analyzing images of the human body 301 captured by a camera (not shown), but may also be detected separately from non-operational actions by a radio wave sensor 1.

[0127] In this way, by detecting the actions of the human body 301 (especially non-manipulated actions) based on the estimated changes in posture, and performing device control according to the detection results, it becomes possible to control the device 200 without any operation on it (operation-less device control).

[0128] (2-4-3) Information acquisition section As shown in Figure 1, the equipment control system 100 further includes an information acquisition unit 226. The information acquisition unit 226 acquires various types of information. These types of information include, for example, environmental information and human identifiers (both described later).

[0129] The information acquisition unit 226 acquires, for example, environmental information. Environmental information refers to information about the environment in which the human body 301 is located. The environment may include, for example, the time of day (daytime, nighttime, early morning, late night, etc.), weather (sunny, rainy, etc.), temperature, humidity, etc., but it may also include the type of room 400 in which the human body 301 is located (bedroom, living room) or its orientation (south-facing, north-facing).

[0130] Furthermore, if multiple people (two or more human bodies 301) may exist in the same environment, the information acquisition unit 226 may acquire a person identifier. Specifically, for example, a set of paired information (a group of paired information) consisting of a person identifier that identifies a person and characteristic information that indicates the characteristics of that person is stored in memory beforehand. The person identifier is, for example, an ID such as "1" or "2," but it can be any information that can identify a person, such as a name or email address. The characteristic information paired with the person identifier is, for example, height information indicating the person's height. However, the characteristic information can also be speed information indicating the person's walking speed, or any information that indicates a characteristic that can distinguish the person from other people. The characteristic information itself may also be used as the person identifier.

[0131] When the posture estimation unit 223 detects a human body 301, the information acquisition unit 226 may acquire characteristic information of the human body 301 from the posture estimation unit 223 and read a person identifier corresponding to the acquired characteristic information from memory.

[0132] The automatic control information described later (see Figure 14) may be customized for each of the multiple person identifiers.

[0133] The behavior detection unit 224 detects non-operational behavior based on the environmental information acquired by the information acquisition unit 226.

[0134] In this way, by utilizing environmental information in addition to changes in posture, it is possible to improve the accuracy of detecting non-manipulative behaviors.

[0135] (2-4-3a) Automatic control and information for automatic control The device control unit 225 controls the device 200 based on non-operational actions using, for example, one or more (in this case, six) automatic control information (a group of automatic control information) as shown in Figure 14. The group of automatic control information is pre-stored in the memory of the control device 2, for example. In this embodiment, the control based on the group of automatic control information is referred to as "automatic control".

[0136] Each of the one or more automatic control information items that make up the automatic control information group includes information about state changes, posture changes, environment, non-operational actions, control content, and control ID, as shown in Figure 14. Specifically, of the six automatic control information items shown in Figure 14, the first automatic control information item includes state change "absent → present", posture change "-", environment "-", non-operational action "entered room", control content "lighting normal on (brightness 5)", and control ID "1". Note that "-" indicates that the information is not included (the same applies hereafter).

[0137] Similarly, the second set of automatic control information includes a state change of "-", a posture change of "standing → sitting", an environment of "daytime", a non-operational action of "desk work", a control content of "brighten the lights (+2)", and a control ID of "2". The third set of automatic control information includes a state change of "-", a posture change of "standing → sitting", an environment of "nighttime", a non-operational action of "relaxing", a control content of "turn on the TV", and a control ID of "3". The fourth set of automatic control information includes a state change of "-", a posture change of "standing or sitting → lying down", an environment of "nighttime", a non-operational action of "going to bed", a control content of "turn off the lights", and a control ID of "4". The fifth set of automatic control information includes a state change of "-", a posture change of "lying down → standing or sitting", an environment of "late at night", a non-operational action of "waking up in the middle of the night", a control content of "dim the lights (brightness 2)", and a control ID of "5". The sixth piece of automatic control information includes a state change "-", a posture change "lying down → standing or sitting", an environment "morning", a non-operational action "getting up", a control content "blind open", and a control ID "6".

[0138] (2-4-3b) Manual control Furthermore, the equipment control unit 225 can also perform control using automatic control information (automatic control) in response to operations on the equipment 200 (hereinafter simply referred to as "operations"). In this embodiment, control based on operations is referred to as "manual control".

[0139] (2-4-4) Accumulation of history and learning The device control system 100 further comprises a history storage unit 227 and a learning unit 228.

[0140] (2-4-4a) History storage unit The history storage unit 227 stores control history information. Control history information refers to information about the control history of the device 200 performed by the device control unit 225. The control history information is, for example, automatic control history information, but it may also be manual control history information. Alternatively, the control history information may include both automatic control history information and manual control history information.

[0141] Automatic control history information refers to information about the control history of device 200 in response to an action (for example, a non-operational action, but it may also be an operational action). The automatic control history information associates the control content of the action-based control with the time the control was performed, environmental information, and a person identifier.

[0142] Manual control history information refers to information regarding the control history of the device 200 in response to an operation. The manual control history information associates the control content of the device 200 based on the operation with one or more pieces of information from among the time the operation was performed, environmental information, and a person identifier.

[0143] When the device control unit 225 controls the device 200, the history storage unit 227 stores control history information, such as that shown in Figure 15, in the memory of the control device 2. Note that the storage location for the control history information may also be an external memory.

[0144] For example, if the equipment control unit 225 has already performed three control operations on the lighting fixture 200a at the time of 9:02, the memory of the control device 2 stores three control history entries, as shown in Figure 15. Each of the three control history entries includes information on the history ID, time, control ID, and operation.

[0145] Of the three control history information entries shown in Figure 15, the first control history information entry includes history ID "1", time "9:00", control ID "1", and operation "-". The second control history information entry includes history ID "2", time "9:01", control ID "2", and operation "-". The third control history information entry includes history ID "3", time "9:02", control ID "-", and operation "Operation to slightly dim the lighting (-1)".

[0146] The aforementioned automatic control history information is the information with "-" for operation information, and the first and second control history information items among the three control history information items above fall into this category. The aforementioned manual control history information is the information with a control ID of "-", and the third control history information item among the three control history information items above falls into this category.

[0147] (2-4-5) Learning Department The learning unit 228 updates the automatic control information when the time and control content indicated by the automatic control history information and the time and operation indicated by the manual control history information satisfy predetermined update conditions.

[0148] (2-4-5a) Update conditions and updates of automatic control information based thereon The renewal conditions are, for example, "that after automatic control based on non-operational behavior is performed on a device, an operation to negate the control content of that automatic control is received within a predetermined time (e.g., 2 minutes)."

[0149] In the example in Figure 15, the time "9:01" and control content (i.e., the control content corresponding to control ID "2" in Figure 14) "Brighten the lights (+2)" shown by the second control history information, the automatic control information, and the time "9:03" and operation "Slightly dim the lights (-1)" shown by the third control history information, the manual control information, satisfy the above update conditions.

[0150] In other words, after an automatic control operation "brighten the lighting (+2)" based on a non-operational action "desk work" is performed on one device "lighting fixture 200a", an operation to cancel the control content of the automatic control operation "dim the lighting slightly (-1)" is received within 2 minutes, so the learning unit 228 determines that the update conditions have been met. Then, the learning unit 228 updates the second automatic control information corresponding to control ID "2" among the six automatic update information in Figure 14 to, for example, "brighten the lighting slightly (+1)".

[0151] Alternatively, the renewal condition could be, for example, "that an operation different from the content of the automatic control has been received a predetermined number of times or more within a predetermined period (for example, 'more than twice a day' or 'more than three times a week')."

[0152] In this way, by accumulating historical information on automatic control in response to non-operational actions (automatic control history information) and historical information on manual control in response to operations, and updating the automatic control information based on these two historical pieces of information when the update conditions are met, a learning function can be realized.

[0153] (3) Specific examples of equipment control systems The equipment control system 100 in this example comprises a radio wave sensor 1 and a control device 2, as shown in Figure 1. The radio wave sensor 1 and the control device 2 each have a communication module and are connected to each other via wired or wireless communication. The control device 2 further includes a processor and memory. Programs and various information are stored in the memory, and the processor operates based on the programs in the memory (further in cooperation with the communication module) to realize the operation of the control device 2 as described later. The radio wave sensor 1 also has a processor and memory, and its transmission and reception operations, as well as communication with the control device 2, may be performed under the control of a program.

[0154] The radio wave sensor 1 is installed on the ceiling of room 400, as shown in Figure 2. The control device 2 is, for example, a multi-remote control, and is installed on the side wall of room 400, as shown in Figure 2.

[0155] The control device 2 estimates the posture (standing, sitting, and lying) of the human body 301 present in the room 400 based on the output of the radio wave sensor 1. Based on the estimated changes in posture, the control device 2 detects the non-operational behavior of the human body 301 and automatically controls multiple devices 200 (lighting fixture 200a, TV 200b, and electric blind 200c) according to the detection results. The control device 2 also has an operating device such as a touch panel and can manually control the multiple devices 200 according to the operation.

[0156] As shown in Figure 2, the control device 2 comprises a reception unit 21, a processing unit 22, and an output unit 23. The processing unit 22 comprises a distance measuring unit 221, a point placement unit 222, a posture estimation unit 223, an action detection unit 224, an equipment control unit 225, an information acquisition unit 226, a history storage unit 227, and a learning unit 228.

[0157] The reception unit 21 receives various types of information. These types of information include, for example, operating information for the device 200. The reception unit 21 may also receive information other than operating information, such as personal information such as the height of a person 301.

[0158] The processing unit 22 performs various processes. These processes include, for example, the processes of the distance measuring unit 221, the point placement unit 222, the attitude estimation unit 223, the action detection unit 224, the equipment control unit 225, the information acquisition unit 226, the history storage unit 227, and the learning unit 228. The processing unit 22 also makes various decisions, as explained in the flowchart.

[0159] More specifically, the processing unit 22 may perform, for example, a holding process, an acquisition process, a location determination process, a human body estimation process, and a placement process. The processing unit 22 performs these processes, for example, each time location-determinable information is output from the radio wave sensor 1, or it may perform them each time a number of location-determinable pieces of information corresponding to one frame is output.

[0160] The retention process is the process of retaining the location-identifiable information output by the radio wave sensor 1. The acquisition process is the process of acquiring the difference between multiple locations-identifiable information that is retained. The location identification process is the process of identifying the location of a moving object (301, 200c) from one or more objects (301, 302, 200c) that constitute the object group (301, 302, 200c) based on the difference, and acquiring location information indicating the identified location. The human body estimation process is the process of estimating whether the acquired location information corresponds to a human body 301 or a moving object other than a human body 301 (for example, an electric blind 200c), at least based on the change in the difference. In this embodiment, the change is usually a change in value such as the difference over time, and may also be called a "time change". However, the change may also be, for example, a change in value according to the position in space (spatial change).

[0161] The placement process is the process of virtually placing points corresponding to the acquired position information in space (three-dimensional space 500) when it is estimated that the acquired position information corresponds to the human body 301. The placement process is performed by the point placement unit 222, which constitutes the processing unit 22.

[0162] In this embodiment, the retention process specifically retains three or more location-identifiable pieces of information. The acquisition process acquires two or more differences with different time differences for the three or more retained location-identifiable pieces of information. The location identification process acquires two or more location pieces corresponding to the two or more acquired differences. The human body estimation process estimates whether each of the two or more acquired location pieces corresponds to the human body 301 or a moving object other than the human body 301 (200c). The placement process virtually places the points corresponding to the location pieces that are estimated to correspond to the human body 301 in space (500).

[0163] Furthermore, the placement process may, based on the estimation results of the human body estimation process, display the points corresponding to the object estimated to be a human body 301 and the points corresponding to the object estimated to be a moving object other than a human body 301 (200c) in a way that makes them distinguishable from each other via the output unit 23 (for example, by displaying them in different colors, different sizes, etc.).

[0164] Furthermore, the processing unit 22 in this embodiment further performs equipment control processing. Equipment control processing is the process of controlling the equipment 200 based at least on a point cloud 501, which is a set of multiple points arranged in space (500). The human body estimation processing estimates the posture or change in posture of the human body 301 based on the distribution of the point cloud 501.

[0165] The processing unit 22 may further perform an action detection process. The action detection process is the process of detecting the actions of a person corresponding to the human body 301 based on the estimated posture or change in posture. The device control process controls the device 200 based on the detected actions, for example.

[0166] The processing unit 22 may further perform information acquisition processing. Information acquisition processing is the process of acquiring environmental information about the environment in which the human body 301 is located. The human body estimation processing further detects behavior based on the environmental information acquired by the information acquisition processing.

[0167] The device control process controls the device 200 based on its actions, for example, using pre-stored automatic control information.

[0168] The processing unit 22 may further perform a history storage process. The history storage process is a process of storing history information relating to the control history of the device 200. The history information may be, for example, automatic control history information. Alternatively, the history information may be manual control history information. The history storage process preferably stores both automatic control history information and manual control history information, but it may also store only one of them.

[0169] The processing unit 22 may further perform a learning process. The learning process is a process that updates the automatic control information when the time and operation indicated by the manual control history information meet predetermined update conditions. Alternatively, the learning process may be a process that updates the automatic control information when the time and control content indicated by the automatic control history information and the time and operation indicated by the manual control history information meet predetermined update conditions.

[0170] Furthermore, the radio wave sensor 1 may perform the conversion process described in "(10) Second Modification of the Equipment Control System," and the processing unit 22 may perform the holding process, difference acquisition process, and motion estimation process described in "(10) Second Modification of the Equipment Control System." In other words, the radio wave sensor 1 may include the conversion unit 11 shown in Figure 16, and the processing unit 22 may include the conversion unit 11, holding unit 12, difference acquisition unit 13, and motion estimation unit 14 shown in Figure 16.

[0171] The output unit 23 outputs various types of information. These types of information include, for example, control information for the device 200 (e.g., remote control signals). The output here is usually transmitted to the device 200, but it may also include display on a screen.

[0172] (4) Example of operation of the equipment control system The control device 2, which constitutes the equipment control system 100, operates, for example, according to the flowcharts shown in Figures 4 to 6.

[0173] (4-1) Overall processing The process shown in Figure 4 is initiated when the equipment control system 100 is started and terminated when it stops operating.

[0174] First, the processing unit 22, which constitutes the control device 2, determines whether or not an FFT result group has been output from the radio wave sensor 1 (step S1). If it is determined that no FFT result group has been output, the process proceeds to step S13.

[0175] If it is determined in step S1 that a group of FFT results has been output, the distance measuring unit 221, which constitutes the processing unit 22, retains the FFT result group (step S2).

[0176] Next, the distance measuring unit 221 determines whether or not a group of FFT results lasting for (K+1) frames or longer has been retained (step S3). If it is determined that a group of FFT results lasting for (K+1) frames or longer has not yet been retained, the process returns to step S1.

[0177] If it is determined in step S3 that a set of FFT results for a period of (K+1) frames or longer is retained, the distance measuring unit 221 performs a one-to-many frame difference acquisition process (step S4). The one-to-many frame difference acquisition process is explained by the flowchart in Figure 5.

[0178] Next, the distance measuring unit 221 acquires L (or K) distance measurement results (group of distance measurement results) corresponding to L (or K) differences (group of differences) (step S5).

[0179] Next, the point placement unit 222 calculates L (or K) three-dimensional coordinates (a group of three-dimensional coordinates) based on the distance measurement results obtained in step S5 (step S6).

[0180] Next, the point placement unit 222 places L (or K) points (point group 501) based on the three-dimensional coordinate group acquired in step S6 into the three-dimensional space 500 (step S7).

[0181] The system determines whether the criteria for initiating the determination have been met (step S8). The criteria for initiating the determination may be, for example, that a predetermined number of points or more are located in the three-dimensional space 500. If it is determined that the criteria for initiating the determination have not yet been met, the process returns to step S1.

[0182] If it is determined in step S8 that the judgment start condition has been met, the attitude estimation unit 223 performs attitude estimation processing (step S9). The attitude estimation processing will be explained with reference to the flowchart in Figure 6.

[0183] Next, the behavior detection unit 224 uses the automatic control information (see Figure 14) to detect non-operational behaviors based on changes in the estimation results in step S9 (e.g., changes in state, changes in posture, environment) (step S10). For example, in response to a change in state from "absent to present," the non-operational behavior "entering the room" is detected. Also, in response to a change in posture from "standing to sitting," it is determined whether the environment (current time of day) is daytime or nighttime, and in the case of daytime, the non-operational behavior "desk work" is detected, while in the case of nighttime, the non-operational behavior "relaxing" is detected.

[0184] Next, the equipment control unit 225 performs equipment control (automatic control) according to the detection results of non-operational actions, etc., in step S10 (step S11). For example, in response to the detection of the non-operational action "entering the room," automatic control is performed to turn on the lighting fixture 200a at normal brightness (brightness 5) (control ID "1" control). Also, in response to the detection of the non-operational action "desk work," automatic control is performed to brighten the lighting fixture 200a (brightness +2) (control ID "2" control). Furthermore, in response to the detection of the non-operational action "relaxing," automatic control is performed to turn on the TV 200b (control ID "3" control).

[0185] Next, the history storage unit 227 stores the automatic control history information related to the automatic control performed in step S11 (step S12). After that, the process returns to step S1.

[0186] If it is determined in step S1 that the FFT results have not yet been output, the processing unit 22 determines whether or not the receiving unit 21 has received an operation request for the device 200 (step S13). If it is determined that the receiving unit 21 has not yet received an operation request for the device 200, the process returns to step S1.

[0187] If the reception unit 21 determines in step S13 that it has received an operation for the device 200, the device control unit 225 performs device control (manual control) corresponding to that operation (step S14). For example, if an operation to slightly dim the lighting fixture 200a (brightness -1) is performed, manual control to slightly dim the lighting fixture 200a is performed.

[0188] Next, the history storage unit 227 stores manual control history information related to the manual control performed in step S14 (step S15).

[0189] Next, the learning unit 228 determines whether the update conditions have been met (step S16). If it is determined that the update conditions have not yet been met, the process returns to step S1.

[0190] If it is determined that the update conditions have been met, the history storage unit 227 updates the information for automatic control (step S17). For example, as shown in Figure 15, one minute after the automatic control of control ID "2", a manual control is performed to slightly dim the lighting fixture 200a, so it is determined that the update conditions have been met, and the control content of the automatic control of control ID "2" is updated to "Brighten the lighting (+1)". After that, the process returns to step S1.

[0191] In the process shown in Figure 4, the radio wave sensor 1 may output three or more IF signals (output signals) corresponding to three or more receiving antennas instead of the FFT result group (location-determinable information), and the processing unit 22 may obtain the FFT conversion result based on the three or more output IF signals. In that case, in step S1 above, the processing unit 22 only needs to perform an FFT conversion on each of the three or more IF signals and determine whether or not an FFT conversion result has been obtained.

[0192] (4-2) One-to-many frame difference acquisition process The one-to-many frame difference acquisition process in step S4 described above is performed, for example, according to the flowchart in Figure 5.

[0193] The processing unit 22 sets the initial value "1" to the variable i (step S41).

[0194] Next, the processing unit 22 determines whether the variable i is greater than K (step S42). If it is determined that the variable i is greater than K, the process proceeds to step S45.

[0195] If it is determined in step S42 that the variable i is not greater than K (i.e., less than or equal to K), the distance measuring unit 221 obtains the difference between the current FFT result group and the i-th preceding FFT result group that is i-th time difference (= i × T seconds) earlier than the current FFT result group, and sets the obtained difference to the variable "i-th difference" (step S43).

[0196] Next, the processing unit 22 increments the variable i (step S44). Then, the process returns to step S42.

[0197] If it is determined in step S42 that the variable i is greater than K, the distance measuring unit 221 selects L differences (for example, three differences ΔA1, ΔA2, and ΔA3: see Figure 11) from the K differences (for example, five differences ΔA1 to ΔA5: only a portion is shown in Figure 11) corresponding to various movements of the human body 301 (step S45). After that, the process returns to the higher-level flowchart (see Figure 4).

[0198] (4-3) Pose estimation process The posture estimation process in step S9 described above is performed, for example, according to the flowchart in Figure 6.

[0199] The attitude estimation unit 223 performs clustering on the point cloud 501 arranged in the three-dimensional space 500 and obtains cluster groups (CL, CL1, CL2: see Figures 12A to 12C) (step S91).

[0200] Next, the attitude estimation unit 223 places the three-dimensional figure (cuboid) 502 (see Figures 13A to 13C) surrounding the cluster group (CL, CL1, CL2) acquired in step S91 into the three-dimensional space 500 (step S92).

[0201] Next, the posture estimation unit 223 obtains the ratio (dimensional ratio) of the length D, width W, and height K of the three-dimensional figure 502 that was positioned in step S92 (step S93).

[0202] Next, the posture estimation unit 223 determines whether the distribution of the cluster group (CL, CL1, CL2) acquired in step S91 (cluster distribution) and the dimensional ratio acquired in step S93 satisfy the supine position condition (step S94).

[0203] If the cluster distribution and dimensional ratio determine that the supine position condition is met, the posture estimation unit 223 sets the variable "posture" to the value "supine position" (step S95). The process then returns to the higher-level flowchart (see Figure 4).

[0204] If the cluster distribution and dimensional ratio are determined not to satisfy the supine position condition, the posture estimation unit 223 further determines whether the cluster distribution and dimensional ratio satisfy the seated position condition (step S96).

[0205] If the cluster distribution and dimensional ratio determine that the seated position conditions are met, the posture estimation unit 223 sets the variable "posture" to the value "seated" (step S97). The process then returns to the higher-level flowchart (see Figure 4).

[0206] If the cluster distribution and dimensional ratio determine that the seated position conditions are not met, the posture estimation unit 223 sets the variable "posture" to the value "standing" (step S98). The process then returns to the higher-level flowchart (see Figure 4).

[0207] (5) Method for estimating human posture, method for controlling equipment, and program The human body posture estimation function of this disclosure may be implemented by a human body posture estimation method or a program. The human body posture estimation method comprises at least steps S2 to S5 (distance measurement steps), steps S6 and S7 (positioning steps), and step S9 (posture estimation step) from among the various steps described above. The program is a program that causes one or more processors to execute the human body posture estimation method.

[0208] The device control function of this disclosure may be implemented by a device control method or a program. The device control method further comprises steps S10 (action detection step) and S11 (device control step), in addition to the distance measurement step, the placement step and the attitude estimation step. The program is a program for causing one or more processors to execute the device control method.

[0209] (6) Variations of non-manipulative behavior Non-manipulative behaviors may include, in addition to going to bed and getting up, falling asleep after going to bed and waking up before getting up. Falling asleep and waking up can be detected, for example, based on changes in body movement (stopping and starting). Stopping body movement may be, for example, a state in which no body movement exceeding a threshold is detected for a predetermined period of time or longer. Starting body movement may be, for example, a state in which body movement exceeding a threshold is detected for a predetermined period of time or longer.

[0210] (7) Variations of the arrangement of each part In this embodiment, the control device 2 comprises a distance measuring unit 221, a point placement unit 222, a posture estimation unit 223, a behavior detection unit 224, an equipment control unit 225, an information acquisition unit 226, a history storage unit 227, and a learning unit 228. However, all of these elements may be provided by the radio wave sensor 1. Alternatively, the radio wave sensor 1 may comprise only the distance measuring unit 221, with the other elements provided by the control device 2. Alternatively, the control device 2 may be built into the radio wave sensor 1.

[0211] (8) First modified example of the equipment control system: Human posture estimation system In this modified example, the device control system 100 shown in Figure 1 omits the elements related to the device control function: the action detection unit 224, the device control unit 225, the information acquisition unit 226, the history storage unit 227, and the learning unit 228. Such a device control system 100 includes at least a distance measuring unit 221, a point placement unit 222, and a posture estimation unit 223, and may be referred to as a "human body posture estimation system 100".

[0212] Furthermore, the human body posture estimation system 100 may also include, in addition to the distance measuring unit 221, point placement unit 222, and posture estimation unit 223, an action detection unit 224, an equipment control unit 225, an information acquisition unit 226, a history storage unit 227, and a learning unit 228.

[0213] (9) Variation of difference acquisition: Intraframe difference The radio wave sensor 1 may perform transmission and reception operations at a rate of N times per frame (where N is an integer of 4 or more), with a predetermined time frame being one frame, and output N pieces of location-identifiable information per frame. The processing unit 22 maintains at least N pieces of location-identifiable information corresponding to one frame. The processing unit 22 then selects three or more pieces of location-identifiable information (representative location-identifiable information) that represent a frame from the N or more pieces of location-identifiable information it maintains that correspond to one or more frames, and obtains two or more differences between the three or more representative location-identifiable information pieces that it has selected. The processing unit 22 then may obtain two or more pieces of location information based on the two or more differences obtained between the three or more representative location-identifiable information pieces that represent one frame.

[0214] In this example, the processing unit 22 further holds N location-identifiable information corresponding to the next frame after the current frame. If two or more location pieces obtained for three or more representative location-identifiable information pieces all correspond to moving objects other than the human body 301 (200c), the processing unit 22 considers the current frame and the next frame as a new frame. Then, the processing unit 22 selects three or more representative location-identifiable information pieces that represent the new frame from the 2 × N location-identifiable information pieces corresponding to the new frame, and obtains two or more differences for the three or more selected representative location-identifiable information pieces. The processing unit 22 may also obtain two or more location pieces based on the two or more differences obtained for the three or more representative location-identifiable information pieces that represent the new frame.

[0215] (10) Second modified example of an instrument control system: Instrument control system including a motion estimation system In this modified example, explanations of matters already mentioned in the embodiment may be omitted or simplified.

[0216] As shown in Figure 16, the device control system 100 in this modified example includes a motion estimation system 100A and a device control device 2B. The motion estimation system 100A comprises a radio wave sensor 1 and a motion estimation device 2A. The motion estimation device 2A comprises a conversion unit 11, a holding unit 12, a difference acquisition unit 13, a motion estimation unit 14, and a point placement unit 222. The motion estimation unit 14 comprises a distance measuring unit 221, a posture estimation unit 223, and a movement detection unit 224. The device control device 2B comprises a device control unit 225, an information acquisition unit 226, a history storage unit 227, and a learning unit 228.

[0217] In the example shown in Figure 16, the motion estimation device 2A includes a conversion unit 11, but the conversion unit 11 may be built into the radio wave sensor 1 or interposed between the radio wave sensor 1 and the motion estimation device 2A. In the embodiment described above, although not shown, the conversion unit 11 is built into the radio wave sensor 1.

[0218] Furthermore, although the motion estimation device 2A and the equipment control device 2B are separate components in the example shown in Figure 16, they may be configured as an integrated unit. In addition, although not shown in Figure 16, the motion estimation system 100A typically includes a reception unit 21, a processing unit 22 (for example, a first processing unit on the motion estimation device 2A side and a second processing unit on the equipment control device 2B side), and an output unit 23 (see Figure 1).

[0219] (10-1) Radio wave sensor The radio wave sensor 1, which constitutes the motion estimation system 100A, transmits a transmitted wave Tr, which is a radio wave modulated in a predetermined manner, toward real space (for example, the inside of room 400 shown in Figure 2; hereinafter referred to as "real space (400)"), receives a reflected wave Re from real space (400), and performs a transmit and receive operation to output an output signal based on the transmitted wave Tr and the reflected wave Re.

[0220] The transmission and reception operations are performed repeatedly at predetermined intervals (i.e., periodically), as in the embodiment, but may also be performed irregularly.

[0221] In the embodiment, the radio wave sensor 1 outputs location-identifiable information such as a group of FFT results, but in this modified example, the radio wave sensor 1 outputs an output signal such as an IF signal. The conversion from the output signal to location-identifiable information is performed outside the radio wave sensor 1 (for example, the conversion unit 11 that constitutes the motion estimation device 2A).

[0222] (10-1-1) Output signal The output signal, for example, in the case of radio waves modulated using the FMCW method, is a signal (IF signal: see Figure 3) that shows the frequency difference Δf between the transmitted wave Tr and the reflected wave Re at the same time. However, the output signal may also be a signal that shows the time difference Δt between the transmitted wave Tr and the reflected wave Re, which can be obtained from the frequency difference Δf. In the case of radio waves modulated using the pulse modulation method, the output signal may also be a signal that shows the time difference Δt between the transmitted wave Tr and the reflected wave Re.

[0223] (10-1-2) Modulation Method The specified method, that is, the radio wave modulation method, is the FMCW method in this example, but it could also be a pulse modulation method, or any method that results in the acquisition of location-determinable information.

[0224] (10-2) Motion Estimation Device The conversion unit 11, which constitutes the motion estimation device 2A, performs a conversion process. The conversion process is the process of converting the output signal output by the radio wave sensor 1 into location-identifiable information. The conversion process is, for example, the process of applying a Fourier transform to the IF signal, which is a type of output signal. The Fourier transform is, for example, FFT, but STFT (short-time Fourier transform) or other methods may also be used.

[0225] Location-identifiable information refers to information that allows for the identification of the position of one or more objects (human body 301, electric blind 200c, stationary object 302) existing in real space (400). For example, this includes, but is not limited to, the FFT result set described in the embodiment.

[0226] The holding unit 12 performs a holding process. In this example, the holding process is the process of holding the position-identifiable information converted by the conversion process. The position-identifiable information is written to, for example, the memory of the motion estimation device 2A and held in the memory for a predetermined period (for example, a 1-frame period, a 3-frame period, etc.).

[0227] The difference acquisition unit 13 performs difference acquisition processing. Difference acquisition processing is the process of acquiring the difference between multiple location-identifiable pieces of information held by the retention processing unit.

[0228] The motion estimation unit 14 performs motion estimation processing. Motion estimation processing is a process for making estimations about moving objects. For example, the motion estimation processing estimates whether each of one or more objects is a moving object or a stationary object (stationary object 302) based on the difference acquired by the difference acquisition processing, identifies the position of the object estimated to be a moving object (human body 301, electric blind 200c), and acquires motion position information indicating the identified position.

[0229] The point placement unit 222 performs point placement processing. Point placement processing is the process of virtually placing points corresponding to the motion position information acquired by the motion estimation processing in a virtual space (for example, the three-dimensional space 500 in the embodiment) corresponding to the real space (400). Point placement processing may be, for example, a multi-point placement process as described in the embodiment as the operation of the point placement unit 222.

[0230] In this example, the difference acquisition process acquires two or more differences with different time differences between the three or more location-identifiable pieces of information held by the retention process. The motion estimation process acquires two or more pieces of motion location information corresponding to the two or more differences acquired by the difference acquisition process. The point placement process virtually places two or more points corresponding to the two or more pieces of motion location information acquired by the motion estimation process in a virtual space (500).

[0231] Here, "virtually placing two or more points in the virtual space (500)" includes cases where "all points" are placed, cases where "only some points are placed" (in other words, at least one point is placed), and even cases where "no points are placed" among the two or more points corresponding to the two or more motion position information obtained by the motion estimation process.

[0232] However, the case where "no points are placed" may be excluded, and in such a point placement process, "all or part" of the two or more points corresponding to the two or more motion position information obtained by the motion estimation process are virtually placed in the virtual space (500). In other words, even if all of the two or more points are determined to be moving objects other than the human body 301, for example, one representative point from the two or more points (any one point, or one point corresponding to the average value of the two or more motion position information corresponding to the two or more points) may be placed.

[0233] Furthermore, if the motion estimation process always places all of the two or more points corresponding to the two or more motion position information obtained, then it becomes unnecessary to determine whether the motion is a human body 301 or another moving object (i.e., whether the motion position information corresponds to the human body 301 or another moving object), as is done in this example.

[0234] In this example, the above judgment is made for each of the two or more acquired motion position information, and if it is determined that the information corresponds to the human body 301, a point corresponding to that motion position information is placed. Therefore, depending on the individual judgment result, it is possible that all of the two or more points corresponding to the two or more motion position information are placed, only some of the points are placed (at least one point is placed), or none of the points are placed.

[0235] In addition, for example, if the motion position information to be judged is information from which it is not expected that a valid judgment result can be obtained, it may be excluded from the judgment, resulting in only some points being placed, or even no points being placed at all. In this example, for example, based on information such as the received intensity of the reflected wave Re corresponding to the motion position information, it may be possible to determine whether or not to include each of the two or more motion position information obtained by the motion estimation process in the above judgment, and to perform the above judgment only on the motion position information that has been determined to be included.

[0236] (10-2-1) Details of point arrangement The point placement process may, for example, place only the points corresponding to the human body 301 among the moving objects (i.e., some of the points of two or more objects). In other words, points corresponding to moving objects other than the human body 301 do not need to be placed. However, points corresponding to moving objects other than the human body 301 (i.e., all of the points of two or more objects) may also be placed.

[0237] Furthermore, the point placement process may, for example, place points corresponding to the human body 301 and other living organisms (such as pets: not shown) among the moving objects. In other words, points corresponding to moving objects other than living organisms may be excluded from the placement process.

[0238] In this way, by acquiring two or more differences with different time differences for three or more location-identifiable pieces of information, an increase in the number of points placed in the virtual space (500) (multiple points in the point cloud 501) can be expected. As a result, it becomes possible to more accurately determine whether the point cloud 501 corresponds to a moving object such as a human body 301 or a stationary object 302, thereby improving the accuracy of estimations related to moving objects using the radio wave sensor 1.

[0239] Furthermore, as a result of improving the accuracy of estimations related to moving objects, the feasibility of controlling the device 200 using the radio wave sensor 1 (for example, the control accuracy of the device control device 2B described later) is improved.

[0240] (10-2-2) Estimation regarding the position information of moving objects: Is the moving object a human body or a moving object other than a human body? The moving object is either the human body 301 or any other moving object. Any other moving object could be, for example, an electrically operated blind 200c that opens and closes electrically, or a cleaning robot that cleans while moving; any moving object is acceptable. The moving object may also include the bodies of other living organisms (for example, pets owned by humans).

[0241] The motion estimation process further estimates, at least based on the difference in changes, whether each of the two or more acquired motion position information points corresponds to the human body 301 or another moving object (200c) other than the human body 301. The point placement process virtually places the points corresponding to the motion position information that the motion estimation process estimated to correspond to the human body 301 in the virtual space (500). In other words, points corresponding to the motion position information that the human body estimation process estimated to correspond to another moving object other than the human body 301 are excluded from placement.

[0242] Thus, in the motion estimation system 100A, when making estimations about the human body 301, by acquiring two or more differences with different time differences for three or more location-identifiable pieces of information, it becomes possible to detect various movements of the human body 301 (for example, body movement, respiratory tremors, limb movements, etc.: see Figure 11), thereby improving the accuracy of estimations about the human body 301.

[0243] (10-2-3) Modified example of estimation regarding moving object position information: Is a moving object other than a human body a living organism or a moving object that is not a living organism? Other moving objects besides the human body 301 may be, for example, living organisms other than the human body 301 (e.g., the body of a living creature such as a pet kept by a person: not shown) or non-living moving objects (e.g., an electric blind 200c).

[0244] In this case, the point placement process places points corresponding to living organisms in the virtual space (500). In other words, points corresponding to moving objects other than living organisms are excluded from placement.

[0245] In this example, the equipment control process (described later) may perform actions such as turning on lighting fixtures or air conditioners when a point cloud 501 corresponding to a living organism is detected (for example, when a person or pet enters the room).

[0246] Furthermore, the motion estimation process may distinguish between a point cloud 501 corresponding to the human body 301 and a point cloud 501 corresponding to a non-human organism, based on the distribution of multiple points in the virtual space (500) (such as the shape of cluster CL).

[0247] Furthermore, the device control process may perform different device control depending on whether the point cloud 501 corresponds to a human body 301 or to a living organism other than a human body 301 (for example, a pet). Specifically, the device control process may, for example, turn on the television when a point cloud 501 corresponding to a human body 301 is detected, while not turning on the television when a point cloud 501 corresponding to a living organism other than a human body 301 (a pet) is detected. In this way, by performing different device control depending on whether the object is a human body 301 or a living organism other than a human body 301, it is possible to improve the comfort of the coexistence environment between humans and pets, and furthermore, to achieve both comfort and energy saving.

[0248] (10-2-4) Estimation considering the received intensity of reflected waves The motion estimation process estimates whether each of the two or more acquired motion position information corresponds to a human body 301 or a moving object other than a human body 301 (200c), based on the difference change, as well as at least one of the received intensity of the reflected wave Re corresponding to the motion position information and the change in the received intensity.

[0249] Thus, by utilizing at least one of the received intensity of the reflected wave Re and the change in received intensity, in addition to the difference, the estimation accuracy regarding whether a moving object (301, 200c) is a human body 301 or a moving object other than a human body 301 (200c) can be improved.

[0250] (10-2-5) Estimation related to point clouds The motion estimation process estimates whether the point cloud 501 as a whole corresponds to the human body 301 or to a moving object other than the human body 301 (200c), based on at least one of the distribution and changes in the distribution of the point cloud 501. In this way, by utilizing at least one of the distribution and changes in the distribution of the point cloud 501 placed in the virtual space (500), the accuracy of the estimation regarding whether a moving object (301, 200c) is the human body 301 or a moving object other than the human body 301 (200c) can be improved.

[0251] (10-2-6) Interframe difference The radio wave sensor 1 performs transmission and reception operations N times per frame (where N is an integer greater than or equal to 1), with a predetermined time frame being one frame. As a result, the radio wave sensor 1 outputs N pieces of location-identifiable information per frame. The retention process retains 3 × N or more pieces of location-identifiable information corresponding to 3 or more frames. The difference acquisition process selects one representative piece of location-identifiable information from each of the 3 or more frames held by the retention process. The difference acquisition process then acquires two or more differences for the three or more representative pieces of location-identifiable information selected. The motion estimation process acquires two or more pieces of motion-identifiable information based on the two or more differences acquired for the three or more representative pieces of location-identifiable information representing each of the 3 or more frames.

[0252] This allows for the selection of three or more representative location-identifiable pieces of information over a period of three or more frames, and the acquisition of two or more differences between the three or more selected representative location-identifiable pieces of information.

[0253] (10-2-7) Intraframe difference The radio wave sensor 1 performs transmission and reception operations at a rate of N times per frame (where N is an integer of 3 or more), with a predetermined time frame defined as one frame, and outputs N or more output signals per frame. The conversion process converts each of the N or more output signals output in one frame into location-identifiable information. The retention process retains at least N pieces of location-identifiable information corresponding to one frame.

[0254] The differential acquisition process selects three or more representative position-specifiable pieces of information representing a frame from among N or more position-specifiable pieces of information corresponding to one or more frames held by the holding process. Then, the differential acquisition process acquires two or more differences for the three or more selected representative position-specifiable pieces of information. The moving object estimation process acquires two or more moving object position pieces of information based on the two or more differences acquired for the three or more representative position-specifiable pieces of information representing one frame.

[0255] As a result, within one frame, three or more representative position-specifiable pieces of information can be selected, and two or more differences can be acquired for the three or more selected representative position-specifiable pieces of information.

[0256] (10-2-7a)Combination of frames The holding process further holds N position-specifiable pieces of information corresponding to the next frame after one frame. When all of the two or more moving object position pieces of information acquired by the moving object estimation process for the three or more representative position-specifiable pieces of information correspond to a moving object (200c) other than the human body 301, the differential acquisition process uses one frame and the next frame as a new frame. Then, the differential acquisition process selects three or more representative position-specifiable pieces of information representing the new frame from among the 2×N position-specifiable pieces of information corresponding to the new frame, and acquires two or more differences for the three or more selected representative position-specifiable pieces of information. The moving object estimation process acquires two or more moving object position pieces of information based on the two or more differences acquired by the differential acquisition process for the three or more representative position-specifiable pieces of information representing the new frame.

[0257] As a result, when two or more differences cannot be acquired within one frame, a new frame can be formed with the next frame, and two or more differences can be acquired within the new frame.

[0258] (10-2-8)Specific example of moving object estimation (10-2-8a)Specification of one-dimensional position by one antenna The radio wave sensor 1 has at least one antenna that receives the reflected wave Re. The output signal is an IF signal generated based on the transmitted wave Tr and the reflected wave Re received by one antenna for one transmission and reception operation. The IF signal is a signal indicating the frequency difference between the transmitted wave Tr and the reflected wave Re, and is generated by mixing the transmitted wave Tr and the reflected wave Re.

[0259] The conversion process is a Fourier transform process. The Fourier transform process is a process of performing a Fourier transform on the IF signal. The position-specifiable information is the Fourier transform result of the Fourier transform process. The holding process holds a plurality of Fourier transform results corresponding to one antenna.

[0260] The moving object estimation process includes a distance measurement process. The distance measurement process in this example is a process of measuring the distance from one antenna to an object estimated to be a moving object (301, 200c) based on the difference between a plurality of Fourier transform results corresponding to one antenna. The moving object position information is one-dimensional position information based on the distance measurement result of the distance measurement process.

[0261] Thereby, in the radio wave sensor 1 using radio waves modulated by the FMCW method, it is possible to improve the accuracy when performing estimation (moving object estimation process) regarding the moving object (301, 200c). Also, even if there is only one antenna that receives the reflected wave Re, it is possible to at least improve the accuracy of distance measurement (specification of one-dimensional position) to the moving object (301, 20)c).

[0262] (10-②-8b) Specification of two-dimensional or three-dimensional position by a plurality of antennas The radio wave sensor 1 has a plurality of antennas that receive the reflected wave Re. From the radio wave sensor 1, a plurality of IF signals corresponding to the plurality of antennas are output for one transmission and reception operation. For each of such a plurality of antennas, the conversion unit 11 performs a conversion process, the holding unit 12 performs a holding process, the difference acquisition unit 13 performs a difference acquisition process, and the moving object estimation unit 14 performs a moving object estimation process including a distance measurement process.

[0263] The location-determinable information is a group of Fourier transform results consisting of multiple Fourier transform results corresponding to multiple antennas. The retention process retains these multiple Fourier transform result groups. The motion estimation process includes multiple distance measurement processes that measure the distance from each of the multiple antennas to an object estimated to be a moving object (301, 200c), based on the differences between each of the multiple Fourier transform result groups for each of the multiple antennas. The motion position information is two-dimensional or three-dimensional position information based on multiple distance measurement results corresponding to the multiple distance measurement processes.

[0264] This improves the accuracy of motion estimation processing, which determines the two-dimensional position (e.g., two-dimensional coordinates in a two-dimensional virtual space 500, or a set of direction and distance, etc.) or three-dimensional position (e.g., three-dimensional coordinates in a three-dimensional virtual space 500, or a set of direction and distance, etc.) of a moving object (301,200c).

[0265] (10-2-9) Motion estimation method and program The motion estimation method in this example comprises at least the following steps from the various steps described in the embodiment: step S1 (conversion step), step S2 (holding step), step S4 (difference acquisition step), steps S5 and S6 (motion estimation steps), and step S7 (point placement step). The program causes one or more processors to execute this motion estimation method.

[0266] (10-3) Equipment control devices The device control unit 225, which constitutes the device control device 2B, performs device control processing to control the device 200 based at least on a point cloud 501, which is a collection of multiple points arranged in a virtual space (500). In this way, by acquiring the point cloud 501 corresponding to moving objects (301, 200c) with the radio wave sensor 1 and controlling the device 200 based on the acquired point cloud 501, it is possible to realize control of the device 200 using the radio wave sensor 1.

[0267] In particular, by acquiring a point cloud 501 corresponding to the human body 301 and controlling the device 200 based on the acquired point cloud 501, it is possible to realize control of the device 200 based on the position and movement of the human body 301.

[0268] (10-3-1) Pose estimation and behavior detection The motion estimation process performed by the motion estimation device 2A includes posture estimation processing. Posture estimation processing is the process of estimating the posture of the human body 301. For example, posture estimation processing estimates the posture of the human body 301 based on the distribution of the point cloud 501 (Figures 12A to 13C). Specifically, posture estimation processing may be the process described using Figure 6 in the embodiment.

[0269] The action detection unit 224 performs action detection processing. In this example, action detection processing is the process of detecting the actions of a person corresponding to the human body 301 based on the posture or change in posture estimated by the posture estimation processing. In this example, the device control processing controls the device 200 based on the actions detected by the action detection processing.

[0270] In this way, by estimating the posture or posture change of the human body 301 based on the distribution of the point cloud 501, detecting the human behavior corresponding to the human body 301 based on the estimation result, and controlling the device 200 based on the detection result, it is possible to realize control of the device 200 based on the posture of the human body 301 and, consequently, on the human behavior.

[0271] (10-3-2) Acquisition of environmental information The information acquisition unit 226 executes information acquisition processing. As mentioned above, information acquisition processing is the process of acquiring environmental information about the environment in which the human body 301 is located. The motion estimation processing further detects actions based on the environmental information acquired by the information acquisition processing.

[0272] In this way, by utilizing environmental information in addition to posture or changes in posture, it is possible to improve the accuracy of behavior detection and, consequently, the control accuracy of the device 200.

[0273] (10-3-3) Equipment control using information for automatic control The machine control process controls the device 200 based on actions using pre-stored automatic control information. Thus, the control of the device 200 based on actions can be automatically performed using the automatic control information.

[0274] (10-3-4) Accumulation of Automatic Control History Information In this example, the history accumulation unit 227 executes a history accumulation process. The history accumulation process in this example accumulates automatic control history information. By accumulating the automatic control history information in this way, learning processing based on the automatic control history information becomes possible, and ultimately, it becomes possible to improve the control accuracy using the automatic control history information. Also, as described above, the automatic control history information associates the control content of the device 200 based on actions with one or more pieces of information among time, environmental information, and person identifier. Thereby, control suitable for an individual person can be performed.

[0275] (10-3-5) Further Accumulation of Manual Control History Information The history accumulation process further accumulates manual control history information. The manual control history information is information regarding the control history of the device 200 based on operations on the device 200. The operations are, for example, operations such as adjusting the brightness of lighting as described above, and are received via an operation device such as a remote control. By further accumulating the manual control history information in this way, it becomes possible to improve the control accuracy using two types of control history information, automatic and manual.

[0276] For example, based on the accumulated automatic control history information and manual control history information, when it is detected that no operation (for example, an operation to dim the lighting) to cancel the control content is performed within a certain time (for example, within 1 minute) after automatic control to brighten the lighting is executed in response to the detection of sitting (hereinafter referred to as "tacit inaction") is detected a predetermined number of times or more (for example, 3 times or more in a week) within a predetermined period, learning processing such as learning that the control content is appropriate is repeated to improve the control accuracy. [[ID=:17]]

[0277] However, the learning process described above can be implemented by accumulating only automatic control history information and not manual control history information. For example, while accumulating automatic control history information, a determination may be made based on real-time operation information as to whether or not tacitly ignored inaction has been detected more than a predetermined number of times (for example, more than three times a week) within a predetermined period, and if tacitly ignored inaction has been detected more than a predetermined number of times within a predetermined period, the learning process that the control content is appropriate may be repeated.

[0278] Alternatively, only manual control history information may be stored, without accumulating automatic control history information. For example, by repeatedly performing a process such as registering the control content "brighten the lights in response to seating detection" in the automatic control information when it is detected that an operation to brighten the lights has been received within a certain time (for example, within 1 minute) of seating detection, and this has been detected more than a predetermined number of times within a predetermined period (for example, more than 3 times a week), it is possible to improve the control accuracy of automatic control of lighting brightness and diversify the content of automatic control.

[0279] Furthermore, as mentioned above, the manual control history information associates the control content of the device 200 based on the operation with one or more pieces of information from among time, environmental information, and person identifier. The learning unit 228, for example, performs a learning process to update the automatic control information when the time and operation indicated by the control history information meet predetermined update conditions. This learning process using automatic control history information makes it possible to enable control that is appropriate to the actions of each individual person.

[0280] (10-3-6) Updating information for automatic control Alternatively, the learning unit 228 may perform a learning process to update the automatic control information when the time and control content indicated by the information and the time and operation indicated by the manual control history information relating to the control history of the device 200 based on the operation on the device 200 satisfy predetermined update conditions. By performing such a learning process using two types of control history information, automatic and manual, it becomes possible to enable control that is suitable for the actions and operations of each individual person.

[0281] (10-3-7) CP Ring In this example, the virtual space (500) is a three-dimensional space 500, similar to that in the embodiment. The motion estimation process obtains a cluster group (CL,CL1,CL2), which is a set of one or more clusters (CL,CL1,CL2), by performing clustering on the point cloud 501, and estimates the posture of the human body 301 based on the distribution of the cluster group (CL,CL1,CL2) in the three-dimensional space 500. This allows for accurate estimation of the posture of the human body 301 based on the distribution of the cluster group (CL,CL1,CL2).

[0282] (10-3-8) Estimation of posture The motion estimation process estimates the posture of the human body 301 based on the ratios of the length, width, and height of the three-dimensional figure 502 surrounding the cluster group (CL, CL1, CL2). This allows for accurate and simple estimation of the posture of the human body 301 based on the ratios of the length D, width W, and height H of the three-dimensional figure 502 surrounding the cluster group (CL, CL1, CL2).

[0283] (10-3-9) Detection of behavior The behavior detection process detects non-operational behaviors, which are actions by the human body 301 that are different from operations on the device 200, based at least on the posture or changes in posture estimated by the posture estimation process. The device control process controls the device 200 based at least on the detected non-operational behaviors. In this way, by controlling the device 200 based on non-operational behaviors (for example, daily activities, which are actions that do not intend to operate the device 200), it is possible to control the device without intentional operation on the device 200 (operation-less device control).

[0284] (10-3-10) Equipment control method and program The device control method uses a radio wave sensor 1 to make estimations regarding moving objects (human body 301, electric blind 200c) and controls the device 200 based on the estimation results. The radio wave sensor 1 transmits a transmission wave Tr, which is a radio wave modulated in a predetermined manner, toward the real space (inside the room 400), receives a reflected wave Re from the real space (400), and performs a transmit / receive operation to output an output signal based on the transmission wave Tr and the reflected wave Re. The device control method comprises a conversion step (S1), a holding step (S2), a difference acquisition step (S4), a moving object estimation step (S5, S6), a point placement step (S7), and a device control step (S11). The conversion step (S1) performs a conversion process to convert the output signal output by the radio wave sensor 1 into location-identifiable information that can identify the position of one or more objects (301, 200c, stationary object 302) present in the real space (400). The retention step (S2) performs retention processing to retain the location-identifiable information converted by the conversion process. The difference acquisition step (S4) performs difference acquisition processing to acquire the difference between multiple location-identifiable information held by the retention process. The motion estimation steps (S5, S6) perform motion estimation processing to estimate whether each of one or more objects is moving or stationary (302) based on the difference acquired by the difference acquisition process, to identify the position of the object estimated to be moving (301, 200c), and to acquire motion position information indicating the identified position. The point placement step (S7) performs point placement processing to virtually place points corresponding to the motion position information acquired by the motion estimation process in a virtual space (500) corresponding to the real space (400). The device control step (S11) controls the device 200 based at least on a point cloud 501, which is a set of multiple points placed in the virtual space (500). The program causes one or more processors to execute this device control method.

[0285] (11) Summary A first aspect of the present disclosure of a device control system (100) comprises a radio wave sensor (1), a conversion unit (11), a holding unit (12), a motion estimation unit (14), a point placement unit (222), and a device control unit (225). The radio wave sensor (1) transmits a transmission wave (Tr), which is a radio wave modulated in a predetermined manner, toward the real space (inside the room 400), receives a reflected wave (Re) from the real space (400), and performs a transmit / receive operation to output an output signal based on the transmitted wave (Tr) and the reflected wave (Re). The conversion unit (11) performs a conversion process to convert the output signal output by the radio wave sensor (1) into location-identifiable information that can identify the position of one or more objects (301, 200c, stationary object 302) present in the real space (400). The holding unit (12) performs a holding process to hold the location-identifiable information converted by the conversion process. The difference acquisition unit (13) performs a difference acquisition process to acquire the difference between multiple location-identifiable pieces of information held by the retention process. The motion estimation unit (14) performs a motion estimation process to estimate whether each of one or more objects is a moving object (302) or a stationary object (301, 200c) based on the difference acquired by the difference acquisition process, to identify the position of the object estimated to be a moving object (301, 200c), and to acquire motion position information indicating the identified position. The point placement unit (222) performs a point placement process to virtually place points corresponding to the motion position information acquired by the motion estimation process in a virtual space (500) corresponding to the real space (400). The equipment control unit (225) performs equipment control processing to control the equipment (200) based at least on a point cloud (501), which is a set of points placed in the virtual space (500).

[0286] According to this embodiment, the radio wave sensor (1) acquires a point cloud (501) corresponding to a moving object (301, 200c), and the device (200) is controlled based on the acquired point cloud (501), thereby enabling the control of the device (200) using the radio wave sensor (1).

[0287] In the equipment control system (100) according to the second embodiment, in the first embodiment, the moving body (301, 200c) is either a human body (301) or a moving object other than a human body (301) (200c). The motion estimation process further estimates, at least based on the difference in changes, whether each of the two or more motion position information acquired with different time differences corresponds to a human body (301) or a moving object other than a human body (301) (200c). The point placement process virtually places points in the virtual space (500) that correspond to the motion position information that the motion estimation process has estimated to correspond to a human body (301) from among the two or more motion position information acquired by the motion estimation process.

[0288] According to this embodiment, a point cloud (501) corresponding to the human body (301) is acquired by the radio wave sensor (1), and the device (200) is controlled based on the acquired point cloud (501), thereby enabling control of the device (200) based on the position and movement of the human body (301).

[0289] In the third embodiment of the device control system (100), in the second embodiment, the motion estimation process includes a posture estimation process that estimates the posture of a human body (301) based on the distribution of a point cloud (501). The device control system (100) further comprises an action detection unit (224). The action detection unit (224) executes an action detection process that detects the actions of a person corresponding to the human body (301) based on the posture or change in posture estimated by the motion estimation process. The device control process controls the device (200) based on the actions detected by the action detection process.

[0290] According to this embodiment, the posture or posture change of a human body (301) is estimated based on the distribution of point clouds (501), human behavior corresponding to the human body (301) is detected based on the estimation result, and the device (200) is controlled based on the detection result, thereby realizing control of the device (200) based on the posture of the human body (301) and, consequently, human behavior.

[0291] The equipment control system (100) according to the fourth embodiment further comprises an information acquisition unit (226) in the third embodiment. The information acquisition unit (226) performs an information acquisition process to acquire environmental information relating to the environment in which a human body (301) is present. The motion estimation process further detects actions based on the environmental information acquired by the information acquisition process.

[0292] According to this embodiment, by utilizing environmental information in addition to posture or posture changes, it is possible to improve the accuracy of behavior detection and, consequently, the control accuracy of the device (200).

[0293] In the equipment control system (100) according to the fifth embodiment, in the fourth embodiment, the equipment control process controls the equipment (200) based on actions using pre-stored automatic control information.

[0294] According to this embodiment, the control of the device (200) based on behavior can be performed automatically using information for automatic control.

[0295] The equipment control system (100) according to the sixth embodiment further comprises a history storage unit (227) in the fifth embodiment. The history storage unit (227) performs a history storage process that stores automatic control history information relating to the control history of equipment based on actions.

[0296] According to this embodiment, by accumulating automatic control history information, it becomes possible to improve control accuracy using automatic control history information.

[0297] In the equipment control system (100) according to the seventh embodiment, in the sixth embodiment, the automatic control history information associates the control content of the equipment (200) based on the action with one or more pieces of information from the time the action was performed, environmental information, and a person identifier that identifies the person.

[0298] According to this embodiment, control tailored to each individual can be performed.

[0299] In the equipment control system (100) according to the eighth embodiment, in the seventh embodiment, the history storage process further stores manual control history information relating to the control history of the equipment (200) based on operations performed on the equipment (200).

[0300] According to this embodiment, by further accumulating manual control history information, it becomes possible to improve control accuracy by utilizing two types of control history information: automatic and manual.

[0301] In the ninth embodiment of the equipment control system (100), in the eighth embodiment, the manual control history information associates the control content of the equipment (200) based on the operation with one or more pieces of information from the time the operation was performed, environmental information, and a person identifier. The equipment control system (100) further comprises a learning unit (228). The learning unit (228) performs a learning process to update the automatic control information when the time and operation indicated by the manual control history information satisfy predetermined update conditions.

[0302] According to this embodiment, learning processing using automatic control history information makes it possible to enable control that is tailored to the behavior of each individual person.

[0303] The equipment control system (100) according to the tenth embodiment further comprises a learning unit (228) in the sixth embodiment. The learning unit (228) performs a learning process to update the automatic control information when the time and control content indicated by the automatic control history information and the time and operation indicated by the manual control history information relating to the control history of the equipment based on the operation on the equipment satisfy predetermined update conditions.

[0304] According to this embodiment, learning processing using two types of control history information, automatic and manual, makes it possible to enable control that is suitable for the actions and operations of each individual person.

[0305] In the 11th embodiment of the device control system (100), in the third embodiment, the virtual space (500) is a three-dimensional space (500). The motion estimation process obtains a cluster group (CL,CL1,CL2), which is a set of one or more clusters (CL,CL1,CL2), by performing clustering on the point cloud (501), and estimates the posture of the human body (301) based on the distribution of the cluster group (CL,CL1,CL2) in the three-dimensional space (500).

[0306] According to this embodiment, the posture of the human body (301) can be accurately estimated based on the distribution of cluster groups (CL, CL1, CL2).

[0307] In the twelfth embodiment of the equipment control system (100), in the eleventh embodiment, the motion estimation process estimates the posture of a human body (301) based on the ratio of the length, width, and height of a three-dimensional figure (502) surrounding a cluster group (CL, CL1, CL2).

[0308] According to this embodiment, the posture of a human body (301) can be estimated accurately and simply based on the ratios of the length (D), width (W), and height (H) of the three-dimensional figure (502) surrounding the cluster group (CL, CL1, CL2).

[0309] In the device control system (100) according to the 13th embodiment, in the third embodiment, the behavior detection process detects non-operational behavior based at least on the posture or change in posture estimated by the posture estimation process. Non-operational behavior is a behavior different from the operation of the human body (301) on the device (200). The device control process controls the device (200) based at least on the detected non-operational behavior.

[0310] According to this embodiment, by controlling the device (200) based on non-operational actions (for example, daily actions that do not intend to operate the device 200), it is possible to control the device (200) without intentional operation (operation-less device control).

[0311] In the 14th embodiment of the device control system (100), in the first embodiment, the difference acquisition process acquires two or more differences with different time differences between three or more position-identifiable pieces of information at different time points that are held by the retention process. The motion estimation process acquires two or more pieces of motion position information corresponding to the two or more differences acquired by the difference acquisition process. The point placement process virtually places two or more points corresponding to the two or more pieces of motion position information acquired by the motion estimation process in a virtual space (500).

[0312] According to this embodiment, by acquiring two or more differences with different time differences for three or more location-identifiable pieces of information, it is expected that the number of points to be placed in the virtual space (500) will increase. As a result, for example, it will be possible to more accurately determine whether a point cloud (501) composed of multiple points placed in the virtual space (500) corresponds to a moving object (301, 200c) such as a human body (301) or to a stationary object (stationary object 302). Therefore, the accuracy of estimation regarding moving objects (301, 200c) using the radio wave sensor (1) can be improved. As a result, the feasibility of controlling the device (200) using the radio wave sensor (1) can be improved.

[0313] The 15th embodiment of the device control method is a device control method that uses a radio wave sensor (1) to make estimations regarding moving objects (human body 301, electric blind 200c) and controls the device (200) based on the estimation results. The radio wave sensor (1) transmits a transmission wave (Tr), which is a radio wave modulated in a predetermined manner, toward the real space (inside the room 400), receives a reflected wave (Re) from the real space (400), and performs a transmit and receive operation to output an output signal based on the transmitted wave (Tr) and the reflected wave (Re). The device control method comprises a conversion step (S1), a holding step (S2), a difference acquisition step (S4), a moving object estimation step (S5, S6), a point placement step (S7), and a device control step (S11). The conversion step (S1) performs a conversion process to convert the output signal output by the radio wave sensor (1) into location-identifiable information that can identify the position of one or more objects (301, 200c, stationary object 302) existing in real space (400). The retention step (S2) performs a retention process to retain the location-identifiable information converted by the conversion process. The difference acquisition step (S4) performs a difference acquisition process to acquire the difference between the multiple location-identifiable information held by the retention process. The motion estimation steps (S5, S6) perform a motion estimation process to estimate whether each of the one or more objects is moving or stationary (302) based on the difference acquired by the difference acquisition process, to identify the position of the object estimated to be moving (301, 200c), and to acquire motion position information indicating the identified position. The point placement step (S7) performs a point placement process to virtually place points corresponding to the motion position information acquired by the motion estimation process in a virtual space (500) corresponding to real space (400). The device control step (S11) controls the device (200) based at least on a point cloud (501), which is a set of multiple points located in a virtual space (500).

[0314] According to this embodiment, similar to the first embodiment, it is possible to realize the control of equipment (200) using a radio wave sensor (1).

[0315] The program according to the 16th embodiment causes one or more processors to execute the device control method described in claim 15.

[0316] According to this embodiment, similar to the first embodiment, it is possible to realize the control of equipment (200) using a radio wave sensor (1). [Explanation of symbols]

[0317] 100. Equipment control systems (human body posture estimation system, motion estimation system) 100A Motion Estimation System 1. Radio wave sensor 11 Conversion section 12 Holding part 13 Difference acquisition part 14 Motion Estimation Unit 2 Control device 2A Motion Estimation Device 2B Equipment control device 21 Reception Department 22 Processing Units 221 Ranging section 222 point placement section 223 Posture estimation section 224 Action Detection Unit 225 Equipment Control Unit 226 Information Acquisition Department 227 History Storage Unit 228 Learning Department 23 Output section 200 equipment 301 Human body 302 Stationary objects 400 rooms (physical space) 500 Three-dimensional space (virtual space) 501 point cloud 502 Solid Figures (Rectangular Prisms) CL, CL1, CL2 cluster group Tr Transmitted Wave Re received wave

Claims

1. A radio wave sensor that performs a transmit and receive operation, transmitting a transmitted wave, which is a radio wave modulated in a predetermined manner, toward real space, receiving a reflected wave from the real space, and outputting an output signal based on the transmitted wave and the reflected wave, A conversion unit performs a conversion process to convert the output signal output by the radio wave sensor into location-identifiable information that can identify the position of each of one or more objects present in the real space. A holding unit that performs a holding process to hold the location-identifiable information converted by the conversion process, A difference acquisition unit performs a difference acquisition process to acquire the difference between a plurality of location-identifiable pieces of information held by the holding process, A motion estimation unit performs motion estimation processing that estimates whether each of the one or more objects is a moving object or a stationary object based on the difference obtained by the difference acquisition process, identifies the position of the object estimated to be a moving object, and acquires motion position information indicating the identified position. A point placement unit performs a point placement process that virtually places points corresponding to the motion position information acquired by the motion estimation process into a virtual space corresponding to the real space, The system comprises an equipment control unit that performs equipment control processing based at least on a point cloud which is a set of points arranged in the virtual space, The moving body is either a human body or a moving object other than a human body. The motion estimation process further estimates, at least based on the change in the difference, whether each of the two or more motion position information pieces obtained with different time differences corresponds to the human body or a moving object other than the human body. The point placement process virtually places points in the virtual space that correspond to the motion position information that the motion estimation process has estimated to correspond to the human body, out of the two or more motion position information acquired by the motion estimation process. Equipment control system.

2. The motion estimation process includes a posture estimation process that estimates the posture of the human body based on the distribution of the point cloud, The system further includes an action detection unit that performs an action detection process to detect human actions corresponding to the human body based on the posture or changes in posture estimated by the posture estimation process, The device control process controls the device based on the action detected by the action detection process. The equipment control system according to claim 1.

3. The information acquisition unit further comprises an information acquisition unit that performs an information acquisition process to acquire environmental information relating to the environment in which the human body is present, The motion estimation process further detects the action based on the environmental information acquired by the information acquisition process. The equipment control system according to claim 2.

4. The device control process controls the device based on the action using pre-stored automatic control information. The equipment control system according to claim 3.

5. The device further comprises a history storage unit that performs a history storage process for storing automatic control history information relating to the control history of the device based on the actions described above. The equipment control system according to claim 4.

6. The automatic control history information associates the control content of the device based on the action with one or more pieces of information from the time the action was performed, the environmental information, and the person identifier that identifies the person. The equipment control system according to claim 5.

7. The history storage process further stores manual control history information relating to the control history of the device based on operations performed on the device. The equipment control system according to claim 6.

8. The manual control history information associates the control content of the device based on the operation with one or more pieces of information from the time the operation was performed, the environmental information, and the person identifier. The system further includes a learning unit that performs a learning process to update the automatic control information when the time and operation indicated by the manual control history information satisfy predetermined update conditions. The equipment control system according to claim 7.

9. A learning unit that further comprises a learning process to update the automatic control information when the time and control content indicated by the automatic control history information and the time and operation indicated by the manual control history information relating to the control history of the device based on the operation of the device satisfy predetermined update conditions. The equipment control system according to claim 5.

10. The virtual space is a three-dimensional space, The motion estimation process obtains a cluster group, which is a collection of one or more clusters, by performing clustering on the point cloud, and estimates the posture of the human body based on the distribution of the cluster group in the three-dimensional space. The equipment control system according to claim 2.

11. The motion estimation process estimates the posture of the human body based on the ratio of the length, width, and height of the three-dimensional figure surrounding the cluster group. The equipment control system according to claim 10.

12. The action detection process detects a non-operational action which is an action of the human body that is different from an operation of the device, based at least on the posture or change in posture estimated by the posture estimation process. The aforementioned device control process controls the device at least based on the detected non-operational behavior. The equipment control system according to claim 2.

13. A radio wave sensor that performs a transmit and receive operation, which transmits a transmitted wave, which is a radio wave modulated in a predetermined manner, toward real space, receives a reflected wave from the real space, and outputs an output signal based on the transmitted wave and the reflected wave, A conversion unit performs a conversion process to convert the output signal output by the radio wave sensor into location-identifiable information that can identify the position of each of one or more objects present in the real space. A holding unit that performs a holding process to hold the location-identifiable information converted by the conversion process, A difference acquisition unit performs a difference acquisition process to acquire the difference between a plurality of location-identifiable pieces of information held by the holding process, A motion estimation unit performs motion estimation processing that estimates whether each of the one or more objects is a moving object or a stationary object based on the difference obtained by the difference acquisition process, identifies the position of the object estimated to be a moving object, and acquires motion position information indicating the identified position. A point placement unit performs a point placement process that virtually places points corresponding to the motion position information acquired by the motion estimation process into a virtual space corresponding to the real space, The system comprises an equipment control unit that performs equipment control processing based at least on a point cloud which is a set of points arranged in the virtual space, The difference acquisition process acquires two or more differences with different time differences between three or more location-identifiable pieces of information at different time points that the retention process has held, The motion estimation process acquires two or more motion position information corresponding to two or more differences acquired by the difference acquisition process, The point placement process virtually places two or more points corresponding to two or more pieces of motion position information with different time differences, obtained by the motion estimation process, into the virtual space. Equipment control system.

14. A device control method that uses a radio wave sensor that performs a transmit and receive operation, which transmits a transmitted wave, which is a radio wave modulated in a predetermined manner, toward real space, receives a reflected wave from the real space, and outputs an output signal based on the transmitted wave and the reflected wave, to make an estimation regarding a moving object, and to control the device based on the estimation result, A conversion step involves performing a conversion process to convert the output signal output by the radio wave sensor into location-identifiable information that can identify the position of each of one or more objects present in the real space. A holding step which includes a holding process that holds the location-identifiable information converted by the conversion process, A difference acquisition step involves performing a difference acquisition process to acquire the difference between a plurality of location-identifiable pieces of information held by the holding process, A motion estimation step involves performing a motion estimation process based on the difference obtained by the difference acquisition process, estimating whether each of the one or more objects is a moving object or a stationary object, identifying the position of the object estimated to be a moving object, and acquiring motion position information indicating the identified position. A point placement step involves performing a point placement process in which points corresponding to the motion position information acquired by the motion estimation process are virtually placed in a virtual space corresponding to the real space, The device control step includes controlling the device based at least on a point cloud which is a set of points arranged in the virtual space, The moving body is either a human body or a moving object other than a human body. The motion estimation process further estimates, at least based on the change in the difference, whether each of the two or more motion position information pieces obtained with different time differences corresponds to the human body or a moving object other than the human body. The point placement process virtually places points in the virtual space that correspond to the motion position information that the motion estimation process has estimated to correspond to the human body, out of the two or more motion position information acquired by the motion estimation process. Equipment control method.

15. A radio wave sensor that performs a transmit / receive operation, which involves transmitting a transmitted wave, which is a radio wave modulated in a predetermined manner, toward real space, receiving a reflected wave from the real space, and outputting an output signal based on the transmitted wave and the reflected wave, is used to make an estimation regarding a moving object, and to control the device based on the estimation result, A conversion step involves performing a conversion process to convert the output signal output by the radio wave sensor into location-identifiable information that can identify the position of each of one or more objects present in the real space. A holding step which includes a holding process that holds the location-identifiable information converted by the conversion process, A difference acquisition step involves performing a difference acquisition process to acquire the difference between a plurality of location-identifiable pieces of information held by the holding process, A motion estimation step involves performing a motion estimation process based on the difference obtained by the difference acquisition process, estimating whether each of the one or more objects is a moving object or a stationary object, identifying the position of the object estimated to be a moving object, and acquiring motion position information indicating the identified position. A point placement step involves performing a point placement process in which points corresponding to the motion position information acquired by the motion estimation process are virtually placed in a virtual space corresponding to the real space, The device control step includes controlling the device based at least on a point cloud which is a set of points arranged in the virtual space, The difference acquisition process acquires two or more differences with different time differences between three or more location-identifiable pieces of information at different time points that the retention process has held, The motion estimation process acquires two or more motion position information corresponding to two or more differences acquired by the difference acquisition process, The point placement process virtually places two or more points corresponding to two or more pieces of motion position information with different time differences, obtained by the motion estimation process, into the virtual space. Equipment control method.

16. To cause one or more processors to execute the device control method according to claim 14 or 15. program.

Images