Information processing system, method for generating teacher data, method for generating trained model, and information processing program

Abstract

To facilitate estimation of an action of an object in a real space.SOLUTION: An information processing system acquires label data indicating temporal change of an action label indicating an action of a sample object in a video showing an object space, acquires CSI data indicating a temporal change of first channel state information about wireless communication in the object space, the first channel state information having time point correspondence with the video, generates teacher data by associating the action label with the first channel state information on the basis of the time point, and outputs the teacher data for generating a learned model which estimates the action of the target object in the object space from second channel state information of the wireless communication in the object space.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] One aspect of the present disclosure relates to an information processing system, a method for generating teacher data, a method for generating a trained model, and an information processing program. [Background technology]

[0002] Methods for estimating the motion of objects in real space are known. For example, Non-Patent Document 1 describes a method for accurately recognizing human activities in videos by a Timeception convolutional layer that determines minute-long temporal patterns. Non-Patent Document 2 describes a method for ensuring diversity of samples showing people's actions by distributing and crowdsourcing the entire video creation process, from script writing to video recording and annotation. Non-Patent Document 3 describes training and testing an Action Transformer model for recognizing and identifying human actions in video clips on the Atomic Visual Actions (AVA) dataset to achieve high performance. Non-Patent Document 4 describes details of the AVA dataset. Non-Patent Document 5 describes a method for recognizing human actions by extracting features of the actions from a data stream of channel state information (CSI) of a commercial Wi-Fi system and building a model and classifier using machine learning. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] N. Hussein, E. Gavves, AWM Smeulders. Timeception for ComplexAction Recognition. arXiv:1812.01289. [Non-Patent Document 2] GA Sigurdsson, G. Varol, X. Wang, A. Farhadi, I Laptev, and A.Gupta. Hollywood in Homes: Crowdsourcing Data Collection for ActivityUnderstanding. arXiv:1604.01753. [Non-Patent Document 3] R. Girdhar, J. Carreira, C. Doersch, and A. Zisserman. Video ActionTransformer Network. arXiv:1812.02707. [Non-Patent Document 4] A. Li, M. Thotakuri, DA Ross, J. Carreira, A. Vostrikov, and A. Zisserman. The AVA-Kinetics Localized Human Actions Video Dataset. arXivpreprint arXiv:2005.00214, 2020. [Non-Patent Document 5] S. Yousefi, H. Narui, S. Dayal, S. Ermon, and S. Valaee, “A surveyon behavior recognition using Wi-Fi channel state information,” IEEECommunications Magazine, vol. 55, no. 10, pp. 98-104, Oct. 2017. Summary of the Invention [Problem to be solved by the invention]

[0004] It is desirable to facilitate the estimation of the motion of objects in real space. [Means for solving the problem]

[0005] An information processing system according to an aspect of the present disclosure includes at least one processor, which acquires label data indicating a time-dependent change in an action label indicating an action of a sample object in an image showing a target space, acquires CSI data indicating a time-dependent change in first channel state information of wireless communication in the target space, the first channel state information having a time correspondence with the image, generates training data by associating the action label with the first channel state information based on time, and outputs the training data to generate a trained model that estimates the action of the target object in the target space from second channel state information of wireless communication in the target space.

[0006] In this aspect, a motion label indicating a motion of a sample object in a video is associated with the first channel state information using the video and the first channel state information that correspond in time. This method automatically labels the channel state information, thereby reducing the effort required to generate training data. As a result, it becomes easy to generate a trained model that estimates the motion of an object from the channel state information, and to estimate the motion of the object using the trained model. Effect of the Invention

[0007] According to one aspect of the present disclosure, it is possible to easily estimate the movement of an object in real space. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of an application of an information processing system. [Diagram 2] FIG. 2 is a diagram illustrating an example of a target space. [Diagram 3] FIG. 2 is a diagram illustrating an example of a functional configuration of an information processing system. [Figure 4] FIG. 2 is a diagram illustrating an example of a hardware configuration of a computer used in the information processing system. [Diagram 5] 11 is a flowchart illustrating an example of an operation of the information processing system. [Figure 6] FIG. 2 is a diagram illustrating an example of a data structure of teacher data. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the description of the drawings, the same or equivalent elements are designated by the same reference numerals, and duplicated explanations will be omitted.

[0010] [System Overview] The information processing system 10 according to the embodiment is a computer system that executes processing related to estimating the motion of an object in a real space. In the present disclosure, an object is a real physical entity that can be visually recognized. The object may be a natural object or an artificial object. The object to be estimated is an object that can move, such as a person, an animal, or a movable product (e.g., a robot, a moving object, etc.). "Estimating the motion of an object" refers to estimating what motion the object actually performed. The real space may be indoors or outdoors.

[0011] In one example, machine learning is used to estimate the behavior of an object. Machine learning is a technique for autonomously finding laws or rules by iteratively learning based on given information. In one example, the information processing system 10 executes machine learning using a machine learning model, which is a computational model including a neural network. A neural network is an information processing model that mimics the mechanism of the human brain and nervous system.

[0012] In machine learning, a machine learning model is trained by repeating learning to obtain a trained model. This corresponds to the learning phase. The trained model generated receives input of channel state information (CSI) of wireless communication and outputs an estimated value indicating the motion of an object. It should be noted that the trained model is a machine learning model estimated to be optimal for estimating the motion of an object, and is not necessarily an "optimal machine learning model in reality". The trained model processes an input vector (input data) indicating the CSI and outputs an estimated value indicating the motion of an object. This corresponds to the estimation phase or operation phase. Trained models are portable between computer systems. Therefore, a trained model generated in one computer system can be used in another computer system. Of course, one computer system may perform both the generation and use of trained models.

[0013] Channel state information (CSI) is information indicating the state of the radio path between a transmitter and a receiver of wireless communication. CSI indicates the amplitude and phase shift of radio waves due to the effects of multipath such as radio wave propagation loss, reflection, and analysis. In one example, the wireless communication is Wi-Fi (registered trademark; the same applies below), a wireless network protocol based on IEEE802.11. CSI indicates the amplitude and phase shift for each of multiple frequency bands (multiple subcarriers). For each subcarrier, the amplitude and phase shift are represented by a matrix of values ​​expressed in terms of the absolute value and argument of a complex number.

[0014] The number of transmitting antennas is N T Let the number of receiving antennas be N R In addition, N obtained from the kth subcarrier is T Let the x-dimensional transmission vector be k Then, N R Let Y be the dimensional receiving vector. k In this case, both vectors X k ,Y k The relationship between them is expressed by the following equation (1). Yk =H k X k +N k …(1) In this formula (1), H k is the N T ×N R is the N-dimensional channel frequency response (CFR) matrix. k is N R It is a dimensional noise vector. If the number of subcarriers is N, then the CSI is N × N T ×N R A multidimensional matrix (N matrices H k ) CSI holds the data for each subcarrier in the form of a matrix, allowing for advanced analysis.

[0015] In one example, the information processing system 10 generates training data used to generate the trained model. The information processing system 10 may generate the trained model by machine learning using the training data. Alternatively, the information processing system 10 may estimate the motion of an object using the trained model.

[0016] FIG. 1 is a diagram showing an example of an application of the information processing system 10 as a processing flow S1. The processing flow S1 includes steps S11 to S15. Steps S11 and S12 are preparations for generating teacher data. Step S13 is generation of teacher data. Step S14 is generation of a trained model using the teacher data (i.e., the learning phase). Step S15 is an estimation process using the trained model (i.e., the operation phase). The information processing system 10 executes at least step S13.

[0017] In step S11, data for generating teacher data is collected. This data includes video data showing the target space and CSI data of wireless communication (e.g., Wi-Fi) in the target space. The video data is obtained by a camera. The CSI data is obtained by periodic data communication between a transmitter and a receiver, for example, by executing a ping command to check network connectivity. The target space refers to a real space in which the object's motion is estimated. The video data indicates video recorded over a long period of time that shows a series of motions of the object (not a single motion but a series of multiple motions). The CSI data indicates changes in CSI over a given time (e.g., a time corresponding to the recording time of the video). Thus, both the video data and the CSI data are time-series data, and indicate changes in the situation of the target space (e.g., the motion of an object in the target space).

[0018] FIG. 2 is a diagram showing a schematic example of a target space. In this example, the target space is a room 80. The room 80 has an entrance 81. A desk 82 and a chair 83 are placed in the room 80. At least one camera 91 is placed to capture the target space (room 80). In one example, the number and positions of the cameras 91 are determined so that the movement of an object (e.g., a person) to be observed in the room 80 can be captured without omission. At least one transmitter 92 and at least one receiver 93 that perform wireless communication (e.g., Wi-Fi) are further placed in the room 80. In the example of FIG. 2, the transmitter 92 and the receiver 93 are placed on either side of the entrance 81. An example of the transmitter 92 is a wireless router. An example of the receiver 93 is a high-function mobile phone (smartphone), a tablet terminal, a laptop computer, and a wireless router for relaying. The receiver 93 may be portable if the environmental conditions are such that a similar change in CSI is observed between the learning phase and the operation phase. In one example, the positions of the transmitter 92 and the receiver 93 are determined so that the CSI changes significantly due to the movement of an object in the target space. In wireless communication, an elliptical wireless communication path called a Fresnel zone is generated with the center being a straight line connecting the transmitter 92 and the receiver 93. In one example, by arranging the transmitter 92 and the receiver 93 so that the Fresnel zone covers the expected movement of an object in the target space, it is expected that the CSI that changes significantly according to the movement can be obtained. If the CSI between the transmitter 92 and the receiver 93 changes significantly due to the movement of the object, the Fresnel zone does not need to be used.

[0019] Returning to FIG. 1, in step S12, a label related to the action of the sample object in the video is set, and label data indicating the label is generated. This label data is time-series data indicating the change in the label over time. The sample object is the subject of the video. A label refers to information that is treated as a correct answer in machine learning. The "label related to the action of the sample object" includes at least an action label indicating the action of the sample object, and may further include an associated label indicating an object related to the action (also referred to as a "related object" in this disclosure). Thus, the label data indicates at least the change in the action label over time, and may further indicate the change in the associated label over time. Examples of related objects include objects carried by a person, objects that can be the purpose or target of a person's action, and the like. The action labels are set, for example, as "no action", "enter", "sit", "stand", "exit", "walk", "run", and "stop". The action label "no action" means that the object does not exist in the target space. The associated labels are set, for example, as "laptop", "smartphone", "desk", "chair", and "door". If there is no related object, the associated label "no related object" may be set.

[0020] In one example, an object itself or the motion of the object is estimated from video data using a trained model generated for video recognition technology or general object recognition, and label data is generated based on the estimation result. For example, the trained model may be a learner based on deep learning, for example, a learner based on a three-dimensional convolutional neural network (3D CNN) such as 3D ResNet. Multiple types of trained models may be used depending on the type of object to be estimated. For example, a trained model for estimating the motion of a sample object and a trained model for identifying a specific object related to the sample object may be used. The label data may be prepared manually in addition to or instead of the trained model. The label data includes at least one label at each time set at a given time interval. In the present disclosure, time is a value indicating a point in the flow of time, and may also be referred to as a time stamp. Time can be expressed using common units such as hours, minutes, and seconds. To indicate the time at an individual date, the time may be further expressed using a year, month, and day.

[0021] Before estimating labels from video data, preprocessing may be performed on the video data. This preprocessing may include data augmentation using techniques such as flipping, grayscaling, blurring, RGB shifting, etc. Alternatively, preprocessing may include frame extraction based on the difference between two grayscaled frames, or cropping of parts directly related to the motion of the sample object.

[0022] In step S13, the information processing system 10 generates training data from the CSI data and the label data. The information processing system 10 identifies a time correspondence between the CSI data and the label data, and generates training data by associating a label with the CSI based on the time. In the present disclosure, the CSI associated with the label is also referred to as "first channel state information (first CSI)." The training data can be said to be data indicating automatically labeled CSI, or data indicating the correspondence between CSI and labels.

[0023] In step S14, a trained model for estimating the object's motion from the CSI is generated by machine learning using the training data. Deep learning may be used to generate the trained model, and for example, a recurrent neural network (RNN) such as a Long Short-Term Memory (LSTM) may be used. For example, step S14 is executed by the information processing system 10 or another computer system.

[0024] In step S15, the motion of the target object is estimated from the CSI using the trained model. The target object is an object whose motion is to be estimated in the operation phase. Step S15 is also executed by the information processing system 10 or another computer system.

[0025] [System configuration] FIG. 3 is a diagram showing an example of a functional configuration of the information processing system 10. In one example, the information processing system 10 includes at least a teacher data generation unit 11 that generates teacher data as a functional module. In one example, the teacher data generation unit 11 includes a label acquisition unit 12, a CSI acquisition unit 13, and a dataset generation unit 14. The label acquisition unit 12 is a functional module that acquires label data. The CSI acquisition unit 13 is a functional module that acquires CSI data. The dataset generation unit 14 is a functional module that generates a dataset (a set of records) of teacher data based on the label data and the CSI data, and outputs the teacher data. "Outputting teacher data" refers to a process of providing teacher data to a functional module in the information processing system 10 or a computer outside the information processing system 10 for subsequent processing.

[0026] In one example, the information processing system 10 further includes a learning unit 15 as a functional module. The learning unit 15 is a functional module that generates a trained model 20 for estimating the motion of an object from the CSI by machine learning using teacher data. In another example, the information processing system 10 further includes an estimation unit 16 as a functional module. The estimation unit 16 is a functional module that estimates the motion of a target object from the CSI using the trained model 20. In the present disclosure, the CSI used to estimate the motion of a target object is also referred to as "second channel state information (second CSI)."

[0027] In one example, the information processing system 10 is connected to a label database 31, a CSI database 32, and a training database 33 via a given communication network. The label database 31 is a device that non-temporarily stores label data. The CSI database 32 is a device that non-temporarily stores CSI data indicating a first CSI. The training database 33 is a device for non-temporarily storing training data. Any of these databases may be part of the information processing system 10, or may be provided in a computer separate from the information processing system 10.

[0028] FIG. 4 is a diagram showing an example of a general hardware configuration of a computer 100 constituting the information processing system 10. For example, the computer 100 includes a processor 101, a main memory unit 102, an auxiliary memory unit 103, a communication control unit 104, an input device 105, and an output device 106. The processor 101 executes an operating system and an application program. The main memory unit 102 is composed of, for example, a ROM and a RAM. The auxiliary memory unit 103 is composed of, for example, a hard disk or a flash memory, and generally stores a larger amount of data than the main memory unit 102. The communication control unit 104 is composed of, for example, a network card or a wireless communication module. The input device 105 is composed of, for example, a keyboard, a mouse, a touch panel, etc. The output device 106 is composed of, for example, a monitor and a speaker.

[0029] Each functional module of the information processing system 10 is realized by a program 110 pre-stored in the auxiliary storage unit 103. Specifically, each functional module is realized by loading the program 110 onto the processor 101 or the main storage unit 102 and having the processor 101 execute the program 110. The processor 101 operates the communication control unit 104, the input device 105, or the output device 106 in accordance with the program 110, and reads and writes data in the main storage unit 102 or the auxiliary storage unit 103. Data or a database required for processing may be stored in the main storage unit 102 or the auxiliary storage unit 103.

[0030] The program 110 may correspond to an information processing program. The program 110 may be provided by being fixedly recorded on a tangible recording medium such as a CD-ROM, a DVD-ROM, or a semiconductor memory. Alternatively, the program 110 may be provided via a communication network as a data signal superimposed on a carrier wave.

[0031] The information processing system 10 may be configured with one computer 100 or may be configured with multiple computers 100. When multiple computers 100 are used, these computers 100 are connected via a communication network such as the Internet or an intranet to logically construct one information processing system 10.

[0032] There is no limitation on the type of computer 100. For example, computer 100 may be a desktop or portable personal computer (PC), a workstation, or a portable terminal such as a highly functional mobile phone (smartphone), a mobile phone, or a personal digital assistant (PDA). Information processing system 10 may be constructed by combining a plurality of types of computers.

[0033] [System Operation] An example of the operation of the information processing system 10 will be described with reference to Fig. 5, and a method for generating teacher data, a method for generating a trained model, and a method for estimating an operation according to the present disclosure will be described. Fig. 5 is a flowchart showing an example of the operation of the information processing system 10 as a processing flow S2. Fig. 5 also shows the correspondence with the processing flow S1.

[0034] In step S21, the label acquiring unit 12 acquires labels estimated from the video. Step S21 corresponds to step S13. In one example, the label acquiring unit 12 accesses the label database 31 to read out label data indicating changes in labels over a given time span. The acquired label data may be represented by one data set (e.g., an integrated data set) or a collection of multiple data sets. Each record of the label data indicates a correspondence between time and at least one label. Each record includes at least one action label indicating an action of a sample object, and may further include an associated label indicating an object present at a position related to the action.

[0035] In step S22, the CSI acquisition unit 13 acquires CSI (first CSI) corresponding to the video. Step S22 corresponds to step S13. In one example, the CSI acquisition unit 13 accesses the CSI database 32 and reads out CSI data indicating the CSI corresponding to the video. Each record of the CSI data indicates a correspondence between time and CSI. "CSI corresponding to the video" refers to CSI that has a time correspondence with the video, that is, CSI measured at the timing when the video was captured.

[0036] When various radio waves fly around in the target space, there is a possibility that radio waves from which CSI is to be obtained may interfere with other radio waves so much that normal CSI (CSI that is meaningful to analyze) cannot be temporarily obtained. The CSI acquisition unit 13 may determine such a time span in which the CSI is considered meaningless as an abnormal section and remove this abnormal section from the CSI data. For example, the CSI acquisition unit 13 may remove from the CSI data an abnormal section in which a situation in which the amplitude fluctuation range is equal to or less than a given threshold (for example, a situation in which the amplitude is a constant value) continues for a given time Th or more. The time Th may be, for example, 1 second, 2 seconds, 3 seconds, 4 seconds, or 5 seconds.

[0037] Steps S21 and S22 mean that the teacher data generating unit 11 acquires the video and CSI recorded in parallel for a given time span. In one example, the "CSI having a time correspondence with the video" is the CSI that is time-synchronized with the video. Alternatively, the "CSI having a time correspondence with the video" may be the CSI that has a known time difference with the video and can be matched with the video taking the difference into account. When time synchronization is performed between the video and the CSI, this process may be performed by the information processing system 10 or by another computer system. The time synchronization may be performed by the Network Time Protocol (NTP) or by an additional or alternative method for more precise time synchronization.

[0038] In step S23, the dataset generation unit 14 generates teacher data by associating a label with the CSI (first CSI) based on the time. Step S23 corresponds to step S13. In one example, the dataset generation unit 14 performs the following process for each record of the CSI data. That is, the dataset generation unit 14 identifies the time indicated by the record, and identifies at least one label indicated in the record of the label data corresponding to the time. Then, the dataset generation unit 14 generates one record of teacher data by associating the time, the CSI, and the label with each other. The dataset generation unit 14 generates multiple records (datasets) of teacher data by associating the label with the CSI for each time in this manner. When processing CSI data from which an abnormal section has been removed, the dataset generation unit 14 performs labeling on the CSI in the sections other than the abnormal section. In one example, the dataset generation unit 14 stores the teacher data in the learning database 33.

[0039] 6 is a diagram showing an example of the data structure of training data. In this example, each record of training data includes a time, a CSI, and a label. In this example, the CSI is the number of antennas of the receiver, N R , the number of transmitter antennas N T , and a multidimensional matrix of complex numbers H, the number of which is the product of the number of subcarriers N per channel (this is the N matrix H k In this example, each record contains one label. As long as the correspondence between time, CSI, and label is shown, the training data may be represented by other data structures.

[0040] Returning to FIG. 5, in step S24, the learning unit 15 uses the teacher data to generate a trained model 20 for estimating the behavior of the target object. Step S24 is a learning phase and corresponds to step S14. As described above, the learning unit 15 may generate the trained model 20 by an RNN such as LSTM. The learning unit 15 executes the following process for each record of the teacher data. The learning unit 15 inputs the CSI indicated by the record to a machine learning model and obtains an estimated value output from the machine learning model. The learning unit 15 updates parameters in the machine learning model using a method such as backpropagation based on the error between the estimated value and the label (i.e., the correct answer) indicated by the record. For example, the learning unit 15 updates the weights of the neural network that constitutes the machine learning model. The learning unit 15 continues machine learning until a given termination condition is met. For example, the learning unit 15 may evaluate the performance of the machine learning model using validation data and terminate the machine learning when the evaluation satisfies a given criterion. Alternatively, the termination condition may be set based on an error, or may be set based on the number of records to be processed, i.e., the number of times learning has been performed.

[0041] In step S25, the estimation unit 16 inputs the CSI (second CSI) in the operation phase into the trained model 20 to estimate the motion of the target object. Step S25 corresponds to step S15. The estimation unit 16 acquires CSI data indicating the CSI in the target space. In the operation phase, the motion of the target object is estimated from the CSI without using an image, so there is no need to acquire an image, much less to make a time correspondence between the CSI and the image. The estimation unit 16 may acquire CSI data input by a user, or may receive CSI data sent from another computer. Alternatively, the estimation unit 16 may access the CSI database 32 or another database to read the CSI data in the operation phase.

[0042] The estimation unit 16 processes CSI data including at least one record. In one example, the estimation unit 16 processes a plurality of records arranged in a time series to estimate a series of movements of a target object in a target space. The estimation unit 16 outputs an estimation result. For example, the estimation unit 16 may display the estimation result on a monitor, may store the estimation result in a predetermined database, or may transmit the estimation result to another computer system. Alternatively, the information processing system 10 may use the estimation result to perform further processing. The estimation unit 16 may output the estimation result in various ways, such as text data, computer graphics (CG), and audio data.

[0043] [effect] As described above, an information processing system according to an aspect of the present disclosure includes at least one processor, which acquires label data indicating a time-dependent change in an action label indicating an action of a sample object in an image showing a target space, acquires CSI data indicating a time-dependent change in the first channel state information of a wireless communication in the target space, the CSI data being first channel state information of the wireless communication in the target space and having a time correspondence with the image, generates training data by associating the action label with the first channel state information based on time, and outputs the training data to generate a trained model that estimates the action of the target object in the target space from second channel state information of the wireless communication in the target space.

[0044] A method for generating teacher data according to one aspect of the present disclosure includes the steps of acquiring label data indicating changes over time in action labels indicating the action of a sample object in an image showing a target space, acquiring first channel state information of wireless communication in the target space, CSI data indicating changes over time in the first channel state information having a time correspondence with the image, generating teacher data by associating the action labels with the first channel state information based on time, and outputting the teacher data to generate a trained model that estimates the action of a target object in the target space from second channel state information of wireless communication in the target space.

[0045] An information processing program according to one aspect of the present disclosure causes a computer to execute the steps of acquiring label data indicating changes over time in action labels indicating the action of a sample object in an image showing a target space, acquiring CSI data indicating changes over time in the first channel state information of wireless communication in the target space, the first channel state information having a time correspondence with the image, generating teacher data by associating the action labels with the first channel state information based on time, and outputting the teacher data to generate a trained model that estimates the action of a target object in the target space from second channel state information of wireless communication in the target space.

[0046] In this aspect, a motion label indicating a motion of a sample object in a video is associated with the first channel state information using the video and the first channel state information that correspond in time. This method automatically labels the channel state information, thereby reducing the effort required to generate training data. As a result, it becomes easy to generate a trained model that estimates the motion of an object from the channel state information, and to estimate the motion of the object using the trained model.

[0047] In order to generate a trained model, it is necessary to prepare a large dataset of training data, and therefore the labor required for labeling also increases. If this labeling is performed manually, a considerable amount of time or effort is spent on preparing the training data, and in addition, there is a high possibility that errors will occur in the labeling. Since the above aspects automate the labeling, large-scale training data can be generated accurately and easily.

[0048] In addition, since the action labels are obtained from the video, it is possible to label different actions of different objects, and therefore different types of action labels can be associated with the first channel state information. This means that the training data can cover different actions of the object. By machine learning using the training data, it becomes possible to estimate different actions of the object from the channel information.

[0049] In an information processing system according to another aspect, at least one processor may generate teacher data by acquiring CSI data indicating first channel state information time-synchronized with the video, and associating an action label with the first channel state information based on the synchronized time. Since time is synchronized between the video and the first channel information, association between the first channel state information and the action label based on the time can be easily performed.

[0050] In the information processing system according to another aspect, at least one processor may generate training data by acquiring label data further indicating a change over time of an associated label indicating an associated object associated with a motion of the sample object, and further associating the associated label with the first channel state information based on time. By using training data in which not only the motion label but also the associated label is associated with the channel state information, it is expected to realize a trained model that can estimate the motion of an object in more detail or more accurately.

[0051] In an information processing system according to another aspect, at least one processor may generate training data by removing an abnormal section from the CSI data, in which the amplitude fluctuation range indicated by the first channel state information is equal to or less than a given threshold for a given time or more, and associating an operation label with the first channel state information indicated by the CSI data from which the abnormal section has been removed. In this abnormal section, the channel state information may be meaningless. By generating training data after removing the abnormal section, it is possible to further improve the accuracy of the trained model.

[0052] In the information processing system according to another aspect, at least one processor may remove, from the CSI data, an abnormal section in which the amplitude is constant for a given time or more. Since channel state information in which the amplitude is constant for a certain time width is obviously meaningless, removing the abnormal section can further improve the accuracy of the trained model.

[0053] In an information processing system according to another aspect, at least one processor may generate a trained model by machine learning using training data.

[0054] A method for generating a trained model according to one aspect of the present invention includes the steps of acquiring label data indicating changes over time in action labels indicating the action of a sample object in an image showing a target space, acquiring CSI data indicating changes over time in the first channel state information of wireless communication in the target space, the first channel state information having a time correspondence with the image, generating teacher data by associating the action labels with the first channel state information based on time, and generating a trained model that estimates the action of a target object in the target space from second channel state information of wireless communication in the target space by machine learning using the teacher data.

[0055] In this aspect, a motion label indicating the motion of a sample object in a video is associated with the first channel state information using a video and first channel state information that correspond in time. This method automatically labels the channel state information, thereby reducing the effort required to generate training data. As a result, a trained model that estimates the motion of an object from the channel state information can be easily generated, and the trained model can be used to easily estimate the motion of the object. The trained model estimates the motion of the object from the channel state information. This means that the estimation is possible without capturing an image of the target space. Therefore, if the object is a person, the motion of the person can be estimated while respecting the person's privacy.

[0056] In the information processing system according to another aspect, at least one processor may input the second channel state information into the trained model to estimate the motion of the target object. In this case, a series of processes including generation of training data, generation of the trained model, and estimation of the motion of the object can be easily performed.

[0057] [Variations] The present disclosure has been described in detail above based on the embodiments. However, the present disclosure is not limited to the above embodiments. The present disclosure can be modified in various ways without departing from the spirit and scope of the present disclosure.

[0058] As described above, the information processing system according to the present disclosure may not have the function of generating the trained model 20, nor may it have the function of estimating the movement of the target object using the trained model 20.

[0059] In this disclosure, the expression "at least one processor executes a first process, executes a second process, ... executes an nth process" or a corresponding expression indicates a concept including a case where the executing entity (i.e., the processor) of n processes from the first process to the nth process changes midway. In other words, this expression indicates a concept including both a case where all n processes are executed by the same processor and a case where the processor changes among the n processes according to an arbitrary policy.

[0060] The processing procedure of the method executed by at least one processor is not limited to the example in the above embodiment. For example, some of the above steps may be omitted, or each step may be executed in a different order. In addition, any two or more of the above steps may be combined, or some of the steps may be modified or deleted. Alternatively, other steps may be executed in addition to the above steps.

[0061] In comparing the magnitude relationship of two numerical values, either of the two criteria "greater than or equal to" or "greater than" may be used, or either of the two criteria "less than or equal to" or "less than." The selection of such criteria does not change the technical meaning of the process of comparing the magnitude relationship of two numerical values. [Explanation of symbols]

[0062] 10...information processing system, 11...teaching data generation unit, 12...label acquisition unit, 13...CSI acquisition unit, 14...dataset generation unit, 15...learning unit, 16...estimation unit, 20...trained model, 31...label database, 32...CSI database, 33...learning database.

Claims

1. At least one processor; the at least one processor: obtaining label data indicating a change over time in an action label indicating an action of a sample object in an image showing the target space; acquiring CSI data indicating a time-dependent change in first channel state information of wireless communication in the target space, the first channel state information having a time correspondence with the image; removing an abnormal section in which the amplitude fluctuation range indicated by the first channel state information is equal to or less than a given threshold for a given time or more from the CSI data; generating teacher data by associating the operation label with the first channel state information indicated by the CSI data from which the abnormal section has been removed, based on the time; outputting the training data to generate a trained model that estimates a motion of a target object in the target space from second channel state information of the wireless communication in the target space; Information processing system.

2. the at least one processor removes from the CSI data the abnormal section in which the amplitude is a constant value for a period of time equal to or longer than the given time; The information processing system according to claim 1 .

3. The at least one processor generates the trained model by machine learning using the training data.

3. The information processing system according to claim 1 or 2.

4. The at least one processor inputs the second channel state information into the trained model to estimate a motion of the target object. The information processing system according to claim 3 .

5. A step of acquiring label data indicating a change over time in an action label indicating an action of a sample object in an image showing a target space; acquiring CSI data indicating a time-dependent change in first channel state information of wireless communication in the target space, the first channel state information having a time correspondence with the image; removing an abnormal section from the CSI data, in which a fluctuation range of an amplitude indicated by the first channel state information is equal to or smaller than a given threshold for a given time or more; generating teacher data by associating the operation label with the first channel state information indicated by the CSI data from which the abnormal section has been removed, based on the time; outputting the training data to generate a trained model that estimates a motion of a target object in the target space from second channel state information of the wireless communication in the target space; A method for generating training data, including:

6. A step of acquiring label data indicating a change over time in an action label indicating an action of a sample object in an image showing a target space; acquiring CSI data indicating a time-dependent change in first channel state information of wireless communication in the target space, the first channel state information having a time correspondence with the image; removing an abnormal section from the CSI data, in which a fluctuation range of an amplitude indicated by the first channel state information is equal to or smaller than a given threshold for a given time or more; generating teacher data by associating the operation label with the first channel state information indicated by the CSI data from which the abnormal section has been removed, based on the time; generating a trained model that estimates a motion of a target object in the target space from second channel state information of the wireless communication in the target space by machine learning using the teacher data; How to generate a trained model including:

7. A step of acquiring label data indicating a change over time in an action label indicating an action of a sample object in an image showing a target space; acquiring CSI data indicating a time-dependent change in first channel state information of wireless communication in the target space, the first channel state information having a time correspondence with the image; removing an abnormal section from the CSI data, in which a fluctuation range of an amplitude indicated by the first channel state information is equal to or smaller than a given threshold for a given time or more; generating teacher data by associating the operation label with the first channel state information indicated by the CSI data from which the abnormal section has been removed, based on the time; outputting the training data to generate a trained model that estimates a motion of a target object in the target space from second channel state information of the wireless communication in the target space; An information processing program that causes a computer to execute the above.

Images