Learning device, estimation device, learning method, and program

Abstract

To reduce data collection cost in imaging using radio waves.SOLUTION: A learning device for learning a model comprises: a storage unit that stores learning data having channel state information and a depth image; and a learning unit that learns the model for estimating the depth image from the channel state information. The model includes: a conversion unit that converts the depth image into a potential vector; a teacher network that includes a reconfiguration unit for reconfiguring the depth image from the potential vector; and a student network that includes the conversion unit for converting the channel state information into the potential network. The learning unit performs learning of the teacher network and learning of the student network.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to wireless sensing technology. [Background technology]

[0002] Wireless sensing, which estimates the surrounding environment by using reflected radio waves, has been actively researched. In particular, imaging using radio waves has been attracting attention as an application of wireless sensing in recent years.

[0003] For example, Non-Patent Document 1 discloses a technique for estimating a camera image from CSI (Channel State Information) by using the image as training data. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Wang, F., Zhou, S., Panev, S., Han, J. and Huang, D.: Personin-WiFi: Fine-Grained Person Perception Using WiFi, 2019 IEEE / CVF International Conference on Computer Vision (ICCV), pp. 5451-5460 (online), 10.1109 / ICCV.2019.00555 (2019). Summary of the Invention [Problem to be solved by the invention]

[0005] However, in conventional techniques such as the technique disclosed in Non-Patent Document 1, imaging using radio waves requires a large amount of learning data to learn a model, and there is a problem in that the cost of collecting the learning data is high. Therefore, the learning process is costly. As a result, the estimation process, which is based on the learning process, is also costly.

[0006] The present invention has been made in consideration of the above-mentioned points, and has an object to provide a technique for realizing imaging using radio waves with lower data collection costs than conventional techniques. [Means for solving the problem]

[0007] According to the disclosed technology, there is provided a learning device for learning a model, comprising: A storage unit for storing training data having channel state information and a depth image; a learning unit configured to learn the model for estimating the depth image from the channel state information; The model is A teacher network including a conversion unit that converts the depth image into a latent vector and a reconstruction unit that reconstructs the depth image from the latent vector; a student network comprising a transformation unit for transforming the channel state information into a latent vector; The learning unit performs learning of the teacher network and learning of the student network. A learning device is provided. Effect of the Invention

[0008] According to the disclosed technology, imaging using radio waves can be achieved with lower data collection costs than conventional technology. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of a system configuration for performing wireless sensing. [Diagram 2] FIG. 1 illustrates an example of the configuration of a learning device 100. [Diagram 3] 4 is a flowchart illustrating the operation of the learning device 100. [Figure 4] FIG. 2 is a diagram illustrating an example of the configuration of an estimation device 200. [Diagram 5] FIG. 1 is a diagram showing a schematic configuration of a model during learning. [Figure 6] FIG. 2 is a diagram showing a detailed configuration of a model. [Figure 7] 2 is a diagram illustrating an example of the configuration of an image generating unit 220 in the estimation device 200. FIG. [Figure 8] FIG. 2 illustrates an example of a hardware configuration of the apparatus. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention (the present embodiment) will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment.

[0011] In the following description, references are represented by numbers such as "[1]", and the names of the references corresponding to the numbers are listed at the end of the specification.

[0012] (Outline of the embodiment) Wireless communication technology is rapidly evolving, and in particular, wireless communication using Wi-Fi (registered trademark) radio waves has become a communications infrastructure that can be used anywhere. In recent years, device-free passive Wi-Fi sensing, which uses Wi-Fi (registered trademark) devices as sensors, has been actively researched as a form of wireless sensing.

[0013] Device-free passive Wi-Fi sensing uses the reflection of Wi-Fi (registered trademark) radio waves to estimate the state of the surrounding environment, so there is no need to attach sensors to the target. In addition, by using existing communication infrastructure, there is no need to install new sensors in the environment. Furthermore, compared to sensing using cameras, it is possible to provide sensing that takes privacy into consideration [1].

[0014] Typical applications of Wi-Fi sensing include crowd counting, event detection, behavior recognition, and input interfaces [2]. These are fundamental technologies for smart homes, monitoring, healthcare, entertainment, games, and more. Among Wi-Fi sensing techniques, imaging using Wi-Fi radio waves has been attracting attention in recent years.

[0015] Imaging technologies including cameras provide visual information that is easily understood by humans and can be applied in a variety of ways. However, sensing using cameras is greatly affected by lighting and the presence or absence of obstructions. On the other hand, Wi-Fi® imaging has the advantage that it can sense deep inside obstructions regardless of the lighting environment. Various researches have been conducted to generate images from propagation information of Wi-Fi® radio waves. Channel State Information (CSI), which is one type of propagation information of Wi-Fi® radio waves, is widely used in research on Wi-Fi® sensing, and the same is true for Wi-Fi® imaging.

[0016] Reference [3] discloses a technology that uses an adjustable reflector array to realize high-resolution imaging. Reference [4] discloses a technology that realizes human 3D skeleton estimation by extracting radio wave reflections from human parts. Reference [5] discloses a technology that extends the technology of Reference [4] to estimate a human 3D mesh using multiple receivers.

[0017] In previous research on Wi-Fi imaging, many studies have been conducted to estimate the image from CSI by using camera images as training data [6], [7] (Non-Patent Document 1). However, it is difficult to estimate the camera image from CSI because the texture and color in the image do not affect the Wi-Fi radio waves.

[0018] Therefore, in this embodiment, we focus on estimating a depth image rather than an RGB image or a grayscale image. Although the texture information of an RGB image or a grayscale image is lost in a depth image, the silhouette information is retained, so that the size, shape, and position of an object can be known.

[0019] In addition, since radio wave propagation changes depending on the 3D shape and position of the object targeted by the depth image, the depth image can be estimated from the CSI using machine learning technology. The above-mentioned existing methods for estimating 3D skeletons and 3D mesh information are specialized for estimating human postures, but the depth images targeted by the technology according to the present embodiment are not limited to humans but are targeted to various objects, making the technology more versatile than the existing methods.

[0020] Depth images are the basis of various real-world applications such as fitness, virtual reality (VR), and monitoring. In this embodiment, the depth image is estimated by focusing on a moving object, which is assumed to be the target of many applications using depth images. However, the scope of application of the technology according to this embodiment is not limited to a moving object.

[0021] Wi-Fi® imaging requires a large amount of Wi-Fi® data and image pair data, which poses the problem of high cost of collecting training data.

[0022] In order to solve the above problems, the depth image generating method according to the present embodiment has the following three features.

[0023] 1) In training a model for estimating depth images from CSI, we use teacher-student learning to train a model consisting of a teacher network that reconstructs depth images and a student network that estimates latent vectors from CSI. This makes it possible to efficiently learn the shape features of objects that are difficult to extract from CSI by using knowledge extracted from depth images by the teacher network.

[0024] 2) Since the teacher network and the student network are separated, the teacher network can be additionally trained using only depth images, for which there is a large amount of public data.

[0025] 3) The use of a Variational Auto-encoder (VAE) enables depth image generation that is robust to data collection conditions. In addition, the loss function incorporates a constraint as an inductive bias that highly similar objects, such as position coordinates and shapes in the depth image, are also close in the latent space, and distance learning is performed. For example, by learning latent representations related to the coordinates of objects in an image, it is expected that images can be generated even if an object exists at position coordinates that are not included in the training data. This can reduce the cost of collecting training data.

[0026] The technology according to the present embodiment having the above-mentioned features can realize a technology for estimating a depth image of an object from CSI. This technology is a technology that has not existed in the past. In addition, it is possible to reduce the cost of collecting pair data of CSI and depth image.

[0027] In this embodiment, it is assumed that Wi-Fi (registered trademark) is used as the wireless system, but the technology according to this embodiment can be applied to any wireless system. For example, the technology according to this embodiment can be applied to wireless systems such as 3G, 4G, 5G, and 6G.

[0028] FIG. 1 shows an example of a basic system configuration for performing wireless sensing in this embodiment.

[0029] 1, the system includes a transmitting device 11 and a receiving device 12. The transmitting device 11 is, for example, a general wireless access point. The receiving device 12 is a terminal that supports wireless communication, such as a laptop PC or a smartphone.

[0030] In Fig. 1, a wireless signal is transmitted from a transmitting device 11 in a space in which a target object exists. The transmitted wireless signal reaches a receiving device 12 in a line of sight from the transmitting device 11, or is reflected by a sensing target object and reaches the receiving device 12. Note that Fig. 1 shows an image in which the target object moves.

[0031] In this embodiment, the receiving device 12 acquires CSI (Channel State Information) from a received wireless signal. An estimation device 200 (to be described later) estimates a depth image of a target object from the CSI. A learning device 100 (to be described later) learns a model for generating a depth image from the CSI.

[0032] CSI indicates amplitude and phase changes due to propagation. In recent years, the Multiple-Input and Multiple-Output (MIMO) technology used in Wi-Fi (registered trademark) communication uses multiple transmitting and receiving antennas. In addition, the Orthogonal Frequency Division Multiplexing (OFDM) method used in Wi-Fi (registered trademark) communication uses multiple subcarriers with different frequencies. It is assumed that OFDM and MIMO are used in the transmitting device 11 and the receiving device 12 in this embodiment as well.

[0033] In this embodiment, for example, CSI is obtained for each packet, each antenna, and each subcarrier. Therefore, CSI is obtained as NP×NT×NR×NS dimensional data. Here, NP, NT, NR, and NS respectively represent the number of packets, the number of transmitting antennas, the number of receiving antennas, and the number of subcarriers.

[0034] Each element of CSI is the channel frequency response (CFR). Since CFR expresses the amplitude and phase changes due to propagation as the absolute value and argument of a complex number, each element of CSI is expressed as follows.

[0035]

number

[0036] First, the related art will be described below, and then the configuration and operation of the apparatus according to the present embodiment will be described in detail.

[0037] (Regarding the related art) <Wi-Fi (registered trademark) sensing> The basic approach of device-free passive Wi-Fi sensing is to estimate the situation of a target using the changes in Wi-Fi (registered trademark) radio waves reflected by moving humans or objects. The movement of the target object affects the propagation of Wi-Fi (registered trademark) radio waves, causing changes in the channel state information (CSI) and the received signal strength indicator (RSSI).

[0038] Since RSSI represents only the strength of the received signal and is obtained as an integer value in dBm units, it is difficult to capture the influence of a moving target object [8].

[0039] By using CSI, the angle-of-arrival (AoA) of the radio wave, the time-of-flight (ToF) of the radio wave, and the Doppler frequency shift (DFS) of the radio wave can be estimated from the phase difference between receiving antennas of the radio wave reflected by the target object, the phase difference between subcarriers, and the phase difference between packets, respectively. Information regarding the movement of the target can be estimated from this information regarding the propagation path.

[0040] For example, Reference [9] discloses a technique for tracking a human position by DFS and AoA. Further, Reference

[10] discloses a technique for estimating a human gesture using DFS. Reference

[11] discloses a technique for human tracking by AoA, ToF, and DFS. Reference

[12] discloses a technique for converting DFS into Body-Coordinate Velocity Profile (BVP), which is an expression useful for action recognition.

[0041] <Imaging by non-visible light> Next, imaging by non-visible light will be described. Techniques for imaging using various media such as infrared rays, ultrasonic waves, and X-rays are known. For example, Reference

[13] discloses a technique for reconstructing a human face by scanning and receiving millimeter waves, and Reference

[14] discloses a skin malignant tumor imaging system based on the dielectric properties of skin tissue. However, since these non-visible light imaging methods require dedicated equipment, they are costly.

[0042] Therefore, a method using Wi-Fi (registered trademark) radio waves as an inexpensive medium has attracted attention. As an example of Wi-Fi (registered trademark) imaging technology, Reference

[15] discloses a technique using a Wi-Fi (registered trademark) device mounted on a drone, and Reference

[16] discloses a technique using a large number of antennas. However, these techniques have the problem that special equipment is required.

[0043] On the other hand, in the technique according to the present embodiment, it is possible to realize the estimation of a depth image using a commercially available Wi-Fi (registered trademark) device.

[0044] <Regarding the use of deep learning in Wi-Fi (registered trademark) sensing> With the popularity of deep learning (DL) technology, many applications of DL to Wi-Fi sensing have been studied. Convolutional Neural Networks (CNN)

[17] and Long-Short Term Memory (LSTM)

[18] are used as feature extractors for spatial and temporal data, respectively, and their applications to CSI have also been widely studied (e.g.,

[19] ,

[20] ,

[21] ).

[0045] Variational Auto-Encoders (VAE) are feature learners that do not require teacher labels

[22] , and their derivative, Beta-VAE, excels at linking the dimensions of latent space to meaningful features

[23] . Conventional CSI-based methods estimate AoA, ToF, and DFS using the MUSIC (MUltiple SIgnal Classification) method for sensing. In contrast, DL model-based methods can automatically extract features useful for a task from CSI. For example, reference

[24] discloses a technology for human skeletal estimation by inputting raw CSI and BVP into a CNN.

[0046] Reference

[19] discloses a human action recognition model using CNN and multi-head attention techniques, which outperforms the baseline in many experiments. Reference

[25] discloses a technique for extracting features through a fully connected layer and estimating 3D poses of multiple people through a convolutional modality conversion network.

[0047] Reference

[20] discloses a technique for implementing gesture recognition using a parallel LSTM-fully connected structure. Reference

[21] discloses a technique for extracting Featured Image Presentation (FIP) from CSI using LSTM, and performing action recognition using CNN and fully connected layers. Based on these techniques, it is expected that the use of DL-based models in Wi-Fi sensing using CSI will exceed the performance of conventional methods without using pre-designed features.

[0048] The device configuration and operation of this embodiment will be described in detail below.

[0049] (Device configuration, operation overview) 2 shows an example of the configuration of the learning device 100 according to this embodiment. As shown in FIG. 2, the learning device 100 includes an input unit 110, a preprocessing unit 115, a learning unit 120, an output unit 130, and a storage unit 140.

[0050] An overview of the operation of the learning device 100 having the above configuration will be described below along the steps of the flowchart in FIG.

[0051] Learning data consisting of pairs of CSI data and depth images is collected using a Wi-Fi (registered trademark) transceiver and a depth camera. In S1 (step 1), the learning data is input to the learning device 100 by the input unit 110. The input learning data is stored in the storage unit 140.

[0052] In S2, the preprocessing unit 115 reads out the learning data from the storage unit 140 and performs preprocessing on each of the CSI and the depth image to extract changes due to moving objects. The preprocessed learning data is stored in the storage unit 140.

[0053] In S3, the learning unit 120 learns a teacher network that reconstructs a depth image using the preprocessed depth image. Specifically, learning means optimizing parameters of a neural network that constitutes the teacher network. The same applies to the student network described later.

[0054] In S4, the learning unit 120 learns a student network that estimates the latent vector of the teacher network from the CSI. When the student network is learned, the parameters of the teacher network are fixed.

[0055] In this embodiment, since the model to be trained is a neural network, training can be performed by backpropagation. That is, the training unit 120 optimizes the parameters of the model by using backpropagation so as to minimize the value of a loss function (a specific example will be described later).

[0056] The learning unit 120 may additionally perform learning of the teacher network using depth images other than the depth images in the learning data.

[0057] Furthermore, in the above example, the learning device 100 performs preprocessing. However, by inputting preprocessed learning data to the learning device 100, the learning device 100 may not perform preprocessing.

[0058] It should be noted that although the learning unit 120 holds a model that is the learning target, the model is not included in the configuration of the learning unit 120. Alternatively, it may be interpreted that the model is included in the configuration of the learning unit 120.

[0059] Through the above processing, the learning unit 120 holds the learned parameters of the teacher network and the student network. The learned parameters are output from the output unit 130 and input to the estimation device 200.

[0060] The estimation device 200 is a device that estimates a depth image from an input CSI by using a model (a student network and a part of a teacher network) in which learned parameters are set. Note that the estimation device 200 and the learning device 100 may be the same device.

[0061] Furthermore, the estimation device 200 may be the receiving device 12 shown in FIG.

[0062] 4 shows an example of the configuration of the estimation device 200. As shown in FIG. 4, the estimation device 200 includes an input unit 210, a preprocessing unit 215, an image generating unit 220, an output unit 230, and a storage unit 240.

[0063] Trained parameters are input from the input unit 210 and stored in the storage unit 240. The image generation unit 220 reads out the trained parameters from the storage unit 240 and sets them in the model, thereby constructing a trained model.

[0064] CSI is input from input unit 210. Preprocessing unit 215 preprocesses the CSI, and the preprocessed CSI is passed to image generation unit 220. Note that input unit 210 may have a function of acquiring CSI from a received wireless signal, or may receive CSI acquired by another device (for example, receiving device 12 in FIG. 1).

[0065] The image generating unit 220 inputs the CSI to the trained model. The image generating unit 220 obtains a depth image as an output from the trained model. The output unit 230 outputs the depth image.

[0066] Each of the above-mentioned processes will now be described in detail.

[0067] (CSI pre-processing) CSI observed from radio waves is affected by various offsets. Specific examples of such offsets include sampling frequency offset (SFO) and carrier frequency offset (CFO). Such offsets cause fluctuations in the phase of CSI. In addition, since the amplitude of the static component of CSI is larger than the amplitude of the dynamic component, it is generally difficult to track a moving object from CSI. To address these problems, in this embodiment, the following two pre-processing steps are performed on CSI. In the following description, it is assumed that pre-processing step 115 performs these pre-processing steps, but in the estimation process, pre-processing step 215 performs these pre-processing steps.

[0068] The pre-processing unit 115 removes offsets using data (CSI) from multiple transmit antennas. Since SFO and CFO are common between transmit antennas, their offsets can be removed by dividing the data from one Tx antenna by that from another Tx antenna. The pre-processing unit 215 then extracts dynamic components from the CSI to extract features related to movement. Here, a high-pass filter is applied to remove static components. In a preliminary experiment conducted by the inventors, it was confirmed that a 5th-order Butterworth filter with a cutoff frequency of 2 Hz works effectively.

[0069] (Pre-processing of depth images) Next, the pre-processing of the depth image will be described. The depth image captured by the depth camera contains the silhouette fluctuation of the target object, missing pixel values, noise, etc. Therefore, in order to stabilize the overall shape of the target object, the pre-processing unit 115 applies a smoothing and filling algorithm to the depth image.

[0070] In this embodiment, it is assumed that a depth image of a moving object, which is expected to be used in many applications, is estimated. Therefore, the preprocessing unit 115 removes a static background from the depth image. Specifically, the preprocessing unit 115 calculates a threshold based on a median value with respect to time for each pixel, and sets depth data greater than the threshold to 0 as background.

[0071] (Network design guidelines) Next, before describing the configuration of a network (also called a model) in this embodiment, first, a design guideline for the network will be described.

[0072] In this embodiment, the task is to obtain a mapping from the CSI domain to the depth image domain, which is called the multimodal alignment task. In this embodiment, a network is designed for this task based on the following guidelines (1) to (3).

[0073] (1) We use a Beta-VAE structure to obtain a regularized mapping from CSI to a depth image. Beta-VAE itself is an existing technique disclosed in

[23] .

[0074] When using a general auto-encoder (AE), the continuity of the encoded latent representation is not guaranteed, and the distance in the latent space is not necessarily related to the similarity of the data. In addition, AE cannot associate each dimension of the latent representation with semantic information such as position and shape.

[0075] On the other hand, Beta-VAE uses variational inference to learn the distribution of input data, thereby giving continuity to the latent space. This allows the decoded image to change smoothly when a sample is moved in the latent space. Furthermore, by increasing the weight of the regularization term in the loss function, each dimension of the latent representation becomes more easily associated with semantic information such as the position and shape of the target.

[0076] (2) It is generally not easy to extract information about the shape of a target object from CSI. Therefore, in this embodiment, a teacher-student structure is used to share semantic information learned from depth images with the CSI domain. A learning method using the teacher-student structure is a learning method that transfers knowledge about inference acquired by a teacher network that has been trained in advance to a student network.

[0077] The teacher network in this embodiment is the aforementioned VAE structure that encodes the depth image into a latent vector and reconstructs the depth image from the latent vector. The student network is a feature extractor that encodes the CSI to match the latent vector obtained by the teacher network. That is, the latent vector acts as a bridge between the domains of CSI and depth image.

[0078] (3) In this embodiment, distance learning is performed using an inductive bias as one of the ideas for reducing the cost of preparing a huge amount of paired data for learning. The inductive bias is a constraint that data with high similarity in semantic information on the depth image is also close in the latent space. As semantic information on the depth image, the shape, size, position coordinates, etc. of the target object obtained from the depth image are used. These pieces of semantic information are quantified and linked as an inductive bias vector. Then, learning is performed so that the similarity between the inductive bias vectors matches the similarity between the latent vectors as teacher data. This is expected to improve generalization to the attributes (size and position) of unknown target objects not included in the learning data.

[0079] (Overview of model during learning) Figure 5 shows a schematic configuration of the model during learning. As shown in Figure 5, this network has a student network 10 and a teacher network 20. The student network 10 has a conversion unit 15 that converts CSI into a latent vector. The teacher network 20 has a conversion unit 21 that converts a depth image into a latent vector, and a reconstruction unit 22 that reconstructs a depth image from the latent vector.

[0080] The transformation unit 15 of the student network 10 has an encoder that receives the CSI as input and outputs parameters of the distribution of the latent vectors, and a reparameterization trick unit that generates the latent vectors from the distribution using the reparameterization trick. Note that the reparameterization trick unit may be external to the transformation unit 15.

[0081] The conversion unit 21 of the teacher network 20 has an encoder that receives a depth image as an input and outputs parameters of a distribution of latent vectors, and a reparameterization trick unit that generates latent vectors from the distribution using a reparameterization trick. Note that the reparameterization trick unit may be outside the conversion unit 21. The reconstruction unit 22 has a decoder that receives a latent vector as an input and outputs a depth image.

[0082] (Details of network configuration during training) The network configuration will now be described in more detail with reference to FIG.

[0083] <Composition of Teacher Network 20> The teacher network 20 uses a Beta-VAE structure to learn the distribution in the latent space of the depth image. In FIG. 6, CNN3 (convolutional layer 3), flattening layer 5, and FC1 (fully connected layer 1) correspond to the encoder in the conversion unit 21 in FIG. 5. Also, in FIG. 6, FC2, flattening layer 6, and CNN4 correspond to the decoder in the reconstruction unit 22 in FIG. 5. Note that, in FIG. 6, for convenience of illustration, the processing contents for learning are shown in the teacher network 20, but the function for learning exists outside the teacher network 20.

[0084] The encoder (CNN3, flattening layer5, and FC1) first takes the depth image y as input and encodes it into a latent representation. At this time, the encoder does not output a latent vector, but outputs the mean μ and standard deviation σ as parameters of the distribution.

[0085] Then, a latent vector z is sampled from the distribution, and the latent vector z is input to the decoder (FC2, flattening layer 6, and CNN 4). The decoder reconstructs a depth image y' from the latent vector z.

[0086] Regarding the above sampling, in fact, we do not sample from a distribution, but rather obtain the latent vector z by using the reparametrization trick to realize learning by the backpropagation method.

[0087] The goal of VAE is to estimate the posterior distribution p(z|y) of the latent vector z given a depth image y. Since it is difficult to compute this directly, we use the neural network encoder q Φ (z|y). The likelihood function for posterior distribution estimation is also approximated by the neural network decoder p θ (y|z), where Φ and θ are the encoder and decoder parameters, respectively. And, q Φ Regularize (z|y) so that it becomes normally distributed.

[0088] The following loss function is used for training Beta-VAE.

[0089]

number

[23] . In this embodiment, the teacher network 20 with a Beta-VAE structure is trained using a weight of β>1. This makes it possible to extract semantic information from the depth image.

[0090] <Learning of teacher network 20 using induction bias> The training unit 120 calculates meaningful physical information, such as the shape, size, and position coordinates of the target object, from the depth image, and vectorizes this information as a recursive bias vector. Then, the following MSE loss is additionally used to compare with the cosine similarity of the latent vectors:

[0091]

number

[0092] c is a vector that concatenates the feature values ​​of the object's size, position coordinates, and shape, calculated from the depth image.

[0093] As the feature of the shape, for example, Histograms of Oriented Gradients (HOG) feature or a latent vector extracted by another neural network such as VAE can be used. Here, the learning unit 120 employs a memory bank method by contrastive learning, and learns to approximate the cosine similarity of z and the cosine similarity of c in the batch t-1 used in the previous parameter update and the batch t which is the current input. This allows the similarity of meaningful information to be reflected in the similarity in the latent space.

[0094] <Student Network 10> Next, a description will be given of the student network 10. CNN1, CNN2, the connection layer 7, and LSTM8 in Fig. 6 correspond to the encoder in the conversion unit 15 in Fig. 5.

[0095] As mentioned above, we use a teacher-student structure to bridge the gap between different domains. During training, the teacher network 20 performs feature extraction in the depth image domain, captures knowledge of this domain as a latent vector z, and passes it on to the student network 10.

[0096] As shown in FIG. 6, the student network 10 has two parallel CNN blocks (CNN1 and CNN2) that receive the amplitude (Amp) and phase (Phase) of the CSI as input, respectively.

[0097] Each CSI sample has dimensions P × R × S, where P is the number of packets, R is the number of receive antennas, and S is the number of subcarriers. These are reshaped into an RS × P matrix to apply the 2D convolution.

[0098] As shown in Figure 6, the outputs of the CNN blocks for amplitude and phase are concatenated and input to LSTM8. The information at the last time in the output sequence of LSTM8 is taken as the latent vector z, and its distribution parameters μ and σ are output. The following knowledge distillation loss is used as the loss function of the student network 10.

[0099]

number

[0100] The first term on the right-hand side of equation (4) is the difference between latent vectors using the MSE loss, and the second term is the KL loss between the distributions of latent vectors. Also, α is the weight of the knowledge distillation loss. Knowledge distillation is commonly used to train smaller networks with knowledge from larger networks

[26] .

[0101] On the other hand, in the task of this embodiment, since it is difficult to learn semantic features related to object positions, etc. directly from CSI, the semantic features learned from the depth image are transferred to the CSI using knowledge distillation.

[0102] (Loss functions and learning methods) The learning unit 120 learns the model according to this embodiment using the following loss function: Beta-VAE loss (the first term on the right side of equation (5)) and MSE loss due to induction bias (the second term on the right side of equation (5)) are used for training the teacher network 20.

[0103]

number

[0104]

number

[0105] The learning unit 120 uses Beta-VAE and an inductive bias vector to learn a latent space that reflects semantic information in the depth image.

[0106] Next, the learning unit 120 trains the student network 10 using the CSI and the latent vectors of the paired depth images. The student network 10 extracts a distribution of latent vectors from the CSI. The learning unit 120 then trains the student network 10 so that the distribution of the latent vectors matches the distribution of the latent vectors in the teacher network 20. The training of the teacher network 20 and the student network 10 can be alternated until the loss converges.

[0107] (Regarding image generating unit 220) As described above, the image generation unit 220 in the estimation device 200 holds a trained model and estimates a depth image from the input CSI using the model.

[0108] An example of the configuration of the image generation unit 220 is shown in Fig. 7. As shown in Fig. 7, the image generation unit 220 includes a conversion unit 15 in the student network 10 and a reconstruction unit 22 in the teacher network 20.

[0109] The conversion unit 15 receives the pre-processed CSI as an input and outputs a latent vector. The latent vector is input to the reconstruction unit 22. The reconstruction unit 22 generates a depth image from the latent vector and outputs it.

[0110] (Hardware configuration example) Any of the devices described in this embodiment (the learning device 100 and the estimation device 200) can be realized, for example, by causing a computer to execute a program. This computer may be a physical computer or a virtual machine on the cloud.

[0111] That is, the device can be realized by executing a program corresponding to the processing performed by the device, using hardware resources such as a CPU and memory built into a computer. The program can be recorded on a computer-readable recording medium (such as a portable memory) and stored or distributed. The program can also be provided via a network such as the Internet or email.

[0112] Fig. 8 is a diagram showing an example of the hardware configuration of the computer. The computer in Fig. 8 has a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, etc., which are mutually connected by a bus BS. The computer may further include a GPU.

[0113] The program for realizing the processing on the computer is provided by a recording medium 1001 such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.

[0114] The memory device 1003 reads out and stores the program from the auxiliary storage device 1002 when an instruction to start the program is received. The CPU 1004 realizes the functions related to the device in accordance with the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network, etc. The display device 1006 displays a GUI (Graphical User Interface) or the like according to a program. The input device 1007 is composed of a keyboard and mouse, buttons, a touch panel, etc., and is used to input various operation instructions. The output device 1008 outputs the results of calculations.

[0115] (Effects of the embodiment) As described above, the present embodiment can realize a technique for estimating a depth image of an object from CSI. Also, the cost of collecting learning data in imaging using radio waves can be reduced.

[0116] The following supplementary notes are further disclosed regarding the above embodiment.

[0117] <Additional Notes> (Additional note 1) A learning device for learning a model, comprising: A storage unit for storing training data having channel state information and a depth image; a learning unit configured to learn the model for estimating the depth image from the channel state information; The model is A teacher network including a conversion unit that converts the depth image into a latent vector and a reconstruction unit that reconstructs the depth image from the latent vector; a student network comprising a transformation unit for transforming the channel state information into a latent vector; The learning unit performs learning of the teacher network and learning of the student network. Learning device. (Additional note 2) The learning unit learns the student network so that a distribution of latent vectors obtained by the student network coincides with a distribution of latent vectors obtained by the trained teacher network. Item 1. A learning device according to item 1. (Additional note 3) A pre-processing section that removes offsets from observed channel state information and static background from depth images captured by a depth camera. 3. The learning device according to claim 1 or 2, further comprising: (Additional note 4) The learning unit learns the teacher network using an inductive bias based on semantic information on the depth image. A learning device according to any one of claims 1 to 3. (Additional note 5) The learning unit learns the teacher network by using a depth image different from the depth image included in the learning data. A learning device according to any one of claims 1 to 4. (Additional note 6) An estimation device for estimating a depth map from channel state information, comprising: An input unit for inputting the channel state information; an image generating unit having the conversion unit in the student network trained by the learning device according to any one of claims 1 to 5 and the reconstruction unit in the teacher network trained by the learning device; The image generator generates the depth image from the channel state information. Estimation device. (Additional note 7) A learning method executed by a learning device including a storage unit for storing learning data having channel state information and a depth image, A learning step of learning a model for estimating the depth image from the channel state information; The model is A teacher network including a conversion unit that converts the depth image into a latent vector and a reconstruction unit that reconstructs the depth image from the latent vector; a student network comprising a transformation unit for transforming the channel state information into a latent vector; In the learning step, the teacher network and the student network are trained. How to learn. (Additional note 8) A non-transitory storage medium storing a program for causing a computer to function as the learning section in the learning device described in any one of claims 1 to 5.

[0118] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and variations are possible within the scope of the gist of the present invention described in the claims. [References] [1] Ohara, K., Maekawa, T. and Matsushita, Y.: Detecting State Changes of Indoor Everyday Objects using Wi-Fi Channel State Information, Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Vol. 1, pp. 1-28 (online), 10.1145 / 3131898 (2017). [2] Khalili, A., Soliman, A.-H., Asaduzzaman, M. and Griffiths, A.: Wi-Fi sensing: applications and challenges, The Journal of Engineering, Vol. 2020, No. 3, pp. 87-97 (online), https: / / doi.org / 10.1049 / joe.2019.0790 (2020). [3] He, Y., Zhang, D. and Chen, Y.: High-Resolution WiFi Imaging With Reconfigurable Intelligent Surfaces, IEEE Internet of Things Journal, Vol. 10, pp. 1775-1786 (2021). [4] Ren, Y., Wang, Z., Tan, S., Chen, Y. and Yang, J.: Winect: 3D Human Pose Tracking for Free-Form Activity Using Commodity WiFi, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 5, No. 4 (online), 10.1145 / 3494973 (2022). [5] Wang, Y., Ren, Y., Chen, Y. and Yang, J.: Wi-Mesh: A WiFi Vision-Based Approach for 3D Human Mesh Construction, Proceedings of the 20th ACM Conference on Embedded Networked Sensor Systems, SenSys '22, New York, NY, USA, Association for Computing Machinery, p. 362-376 (online), 10.1145 / 3560905.3568536 (2023). [6] Kefayati, M. H., Pourahmadi, V. and Aghaeinia, H.: Multi-view WiFi imaging, Signal Processing, Vol. 197, p. 108552 (online), https: / / doi.org / 10.1016 / j.sigpro.2022.108552 (2022). [7] Wang, F., Zhou, S., Panev, S., Han, J. and Huang, D.: Personin-WiFi: Fine-Grained Person Perception Using WiFi, 2019 IEEE / CVF International Conference on Computer Vision (ICCV), pp. 5451-5460 (online), 10.1109 / ICCV.2019.00555 (2019). [8] Yang, Z., Zhou, Z. and Liu, Y.: From RSSI to CSI: Indoor Localization via Channel Response, ACM Comput. Surv., Vol. 46, No. 2 (online), 10.1145 / 2543581.2543592 (2013). [9] Li, X., Zhang, D., Lv, Q., Xiong, J., Li, S., Zhang, Y. and Mei, H.: IndoTrack: Device-Free Indoor Human Tracking with Commodity Wi-Fi, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 1, No. 3 (online), 10.1145 / 3130940 (2017).

[10] Pu, Q., Gupta, S., Gollakota, S. and Patel, S.: Whole-Home Gesture Recognition Using Wireless Signals, Proceedings of the 19th Annual International Conference on Mobile Computing amp; Networking, MobiCom '13, New York, NY, USA, Association for Computing Machinery, pp. 27-38 (online), 10.1145 / 2500423.2500436 (2013).

[11] Qian, K., Wu, C., Zhang, Y., Zhang, G., Yang, Z. and Liu, Y.: Widar2.0: Passive Human Tracking with a Single Wi-Fi Link, Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services, MobiSys '18, New York, NY, USA, Association for Computing Machinery, pp. 350-361 (online), 10.1145 / 3210240.3210314 (2018).

[12] Zhang, Y., Zheng, Y., Qian, K., Zhang, G., Liu, Y., Wu, C. and Yang, Z.: Widar3.0: Zero-Effort Cross-Domain Gesture Recognition With Wi-Fi, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 44, No. 11, pp. 8671-8688 (online), 10.1109 / TPAMI.2021.3105387 (2022).

[13] Xie, J., Kong, H., Yu, J., Chen, Y., Kong, L., Zhu, Y. and Tang, F.: Mm3DFace: Nonintrusive 3D Facial Reconstruction Leveraging MmWave Signals, Proceedings of the 21st Annual International Conference on Mobile Systems, Applications and Services, MobiSys'23, New York, NY, USA, Association for Computing Machinery, p. 462-474 (online), 10.1145 / 3581791.3596839 (2023).

[14] Mirbeik, A., Ashinoff, R., Jong, T., Aued, A. and Tavassolian, N.: Real-time high-resolution millimeter-wave imaging for in-vivo skin cancer diagnosis, Scientific Reports, Vol. 12, No. 1 (online), 10.1038 / s41598-022-09047-6.

[15] Karanam, C. R. and Mostofi, Y.: 3D Through-Wall Imaging with Unmanned Aerial Vehicles Using WiFi, 2017 16th ACM / IEEE International Conference on Information Processing in Sensor Networks (IPSN), pp. 131-142 (2017).

[16] Zhao, M., Li, T., Alsheikh, M. A., Tian, Y., Zhao, H., Torralba, A. and Katabi, D.: Through-Wall Human Pose Estimation Using Radio Signals, 2018 IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp. 7356-7365 (online), 10.1109 / CVPR.2018.00768 (2018).

[17] LeCun, Y., Bengio, Y. and Hinton, G.: Deep learning, Nature, Vol. 521, No. 7553, pp. 436-444 (2015).

[18] Hochreiter, S. and Schmidhuber, J.: Long Short-Term Memory, Neural Computation, Vol. 9, No. 8, pp. 1735-1780 (1997).

[19] Li, B., Cui, W., Wang, W., Zhang, L., Chen, Z. and Wu, M.: Two-Stream Convolution Augmented Transformer for Human Activity Recognition, AAAI Conference on Artificial Intelligence (2021).

[20] Tang, Z., Liu, Q., Wu, M., Chen, W. and Huang, J.: WiFi CSI gesture recognition based on parallel LSTM-FCN deep space-time neural network, China Communications, Vol. 18, No. 3, pp. 205-215 (online), 10.23919 / JCC.2021.03.016 (2021).

[21] Guo, L., Zhang, H., Wang, C., Guo, W., Diao, G., Lu, B., Lin, C. and Wang, L.: Towards CSI-based diversity activity recognition via LSTM-CNN encoder-decoder neural network, Neurocomputing, Vol. 444, pp. 260-273 (online), https: / / doi.org / 10.1016 / j.neucom.2020.02.137 (2021).

[22] Kingma, D. P. and Welling, M.: Auto-Encoding Variational Bayes, CoRR, Vol. abs / 1312.6114 (2013).

[23] Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S. and Lerchner, A.: beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework, International Conference on Learning Representations, (online), https: / / openreview.net / forum?id=Sy2fzU9gl (2017).

[24] Jiang, W., Xue, H., Miao, C., Wang, S., Lin, S., Tian, ​​C., Murali, S., Hu, H., Sun, Z. and Su, L.: Towards 3D Human Pose Construction Using Wifi, Proceedings of the 26th Annual International Conference on Mobile Computing and Networking, MobiCom '20, New York, NY, USA, Association for Computing Machinery, (online), 10.1145 / 3372224.3380900 (2020).

[25] Geng, J., Huang, D. and la Torre, FD: DensePose From WiFi, ArXiv, Vol. abs / 2301.00250 (2022).

[26] Hinton, G.E., Vinyals, O. and Dean, J.: Distilling the Knowledge in a Neural Network, ArXiv, Vol. abs / 1503.02531 (2015). [Explanation of symbols]

[0119] 10. Student Network 11 Transmitting device 12 Receiving device 15 Conversion section 20 Teacher Network 21 Conversion section 22 Reconstruction part 100 Learning Device 110 Input section 115 Pretreatment section 120 Learning Department 130 Output section 140 Storage section 200 Estimation device 210 Input section 215 Pretreatment section 220 Image Generation Unit 230 Output section 240 Storage section 1000 Drive Device 1001 Recording media 1002 Auxiliary storage 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input Device 1008 Output device

Claims

1. A learning device for learning a model, comprising: A storage unit for storing training data having channel state information and a depth image; a learning unit configured to learn the model for estimating the depth image from the channel state information; The model is A teacher network including a conversion unit that converts the depth image into a latent vector and a reconstruction unit that reconstructs the depth image from the latent vector; a student network comprising a transformation unit for transforming the channel state information into a latent vector; The learning unit performs learning of the teacher network and learning of the student network. Learning device.

2. The learning unit learns the student network so that a distribution of latent vectors obtained by the student network coincides with a distribution of latent vectors obtained by the trained teacher network. The learning device according to claim 1 .

3. A pre-processing section that removes offsets from observed channel state information and static background from depth images captured by a depth camera. The learning device according to claim 1 , further comprising:

4. The learning unit learns the teacher network using an inductive bias based on semantic information on the depth image. The learning device according to claim 1 .

5. The learning unit learns the teacher network by using a depth image different from the depth image included in the learning data. The learning device according to claim 1 .

6. An estimation device for estimating a depth map from channel state information, comprising: An input unit for inputting the channel state information; an image generating unit having the conversion unit in the student network trained by the learning device according to any one of claims 1 to 5 and the reconstruction unit in the teacher network trained by the learning device; The image generator generates the depth image from the channel state information. Estimation device.

7. A learning method executed by a learning device including a storage unit for storing learning data having channel state information and a depth image, A learning step of learning a model for estimating the depth image from the channel state information; The model is A teacher network including a conversion unit that converts the depth image into a latent vector and a reconstruction unit that reconstructs the depth image from the latent vector; a student network comprising a transformation unit for transforming the channel state information into a latent vector; In the learning step, the teacher network and the student network are trained. How to learn.

8. A program for causing a computer to function as the learning unit in the learning device according to any one of claims 1 to 5.

Images