Information processing device, information processing method, and information processing program

Abstract

The location information and sensor data are embedded and transmitted. [Solution] An information processing device includes a first acquisition unit that acquires location information, a second acquisition unit that acquires sensor data from a sensor, a conversion unit that converts the location information and sensor data into vectors, and a transmission unit that transmits the vectors. In this case, the conversion unit converts multiple pieces of location information and sensor data into a single vector all at once. The conversion unit converts the location information and sensor data into vectors using a vector conversion model obtained by machine learning or in accordance with predetermined conversion rules.

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] A technique for transmitting location information of a communication device with low power consumption has been disclosed. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-042681 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the above conventional technology, the manually operated vehicle merely transmits communication device information, including information on the current location, to the communication partner at a predetermined transmission interval by low power wide area wireless communication (LPWA). Normally, the communication device does not consume much power when it only acquires location information, but consumes a large amount of power when the communication device transmits location information to the outside (especially when moving at high speed). Even if the power consumption per transmission is reduced, the power consumption increases depending on the number of times the location information is repeatedly transmitted. A manually operated vehicle as exemplified in the above conventional technology may not have a shortage of power capacity, but it is difficult to say that the power capacity is sufficient for a terminal device (handheld device) such as a smartphone.

[0005] In recent years, from the viewpoint of protecting personal information, there has been a tendency for the specifications of terminal devices such as smartphones to place stricter restrictions on the transmission of location information of personal terminal devices to the outside, making it extremely difficult for businesses to obtain location information of personal terminal devices one by one. For example, the option to "always allow" the use of location information of personal terminal devices is disappearing. In addition, there is a problem in that location information cannot be transmitted when an application is running in the background due to control on the OS (Operating System) side.

[0006] Furthermore, in terminal devices such as smartphones manufactured in recent years, it is possible to acquire not only location information but also sensor data such as environmental sound, acceleration, and atmospheric pressure. By acquiring such sensor data, it becomes easier to estimate the user's environment, and it becomes possible to provide services more suited to the user. However, there is also a problem that such sensor data cannot be acquired by businesses due to the Personal Information Protection Act and other regulations.

[0007] The present application has been made in consideration of the above, and has an object to embed and transmit location information and sensor data. [Means for solving the problem]

[0008] The information processing device according to the present application comprises: An information processing device used as a user's terminal device, Location information Raw data A first acquisition unit that acquires Regarding the user's situation and surroundings A second acquisition unit that acquires sensor data; Raw data and the sensor data, and Raw data and the sensor data set A vector space of user actions The embedding is performed, and the position information Raw data and based on the sensor data Indicate the user's actions The present invention is characterized by comprising a conversion unit that converts the signal into an embedding vector, and a transmission unit that transmits the embedding vector. Effect of the Invention

[0009] According to one aspect of the embodiment, the location information and the sensor data can be embedded and transmitted. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is an explanatory diagram showing an overview of an information processing method according to an embodiment. [Diagram 2] FIG. 2 is an explanatory diagram showing an overview of a method for predicting future data according to an embodiment. [Diagram 3] FIG. 3 is a diagram illustrating an example of the configuration of an information processing system according to the embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of the configuration of a terminal device according to the embodiment. [Diagram 5] FIG. 5 is a diagram illustrating an example of the configuration of the information processing device according to the embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of the conversion information database. [Figure 7] FIG. 7 is a diagram illustrating an example of the vector information database. [Figure 8] FIG. 8 is a sequence diagram showing a processing procedure according to the embodiment. [Figure 9] FIG. 9 is a diagram illustrating an example of a hardware configuration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, the information processing device, the information processing method, and the information processing program according to the present application will be described in detail with reference to the drawings. Note that the information processing device, the information processing method, and the information processing program according to the present application are not limited to the embodiments. In addition, the same components in the following embodiments are denoted by the same reference numerals, and duplicated descriptions are omitted.

[0012] [1. Overview of information processing method] First, an overview of an information processing method performed by an information processing device according to an embodiment will be described with reference to Fig. 1. Fig. 1 is an explanatory diagram showing an overview of an information processing method according to an embodiment. Note that Fig. 1 describes an example in which a terminal device embeds location information and sensor data and transmits them.

[0013] 1, the information processing system 1 includes a terminal device 10 and an information providing device 100. The terminal device 10 and the information providing device 100 are connected to each other via a network N (see FIG. 3) in a wired or wireless manner so as to be able to communicate with each other.

[0014] The terminal device 10 is a smart device such as a smartphone or tablet used by a user U (user), and is a mobile terminal device capable of communicating with any server device via a wireless communication network such as 4G (Generation) or LTE (Long Term Evolution). The terminal device 10 has a screen such as a liquid crystal display having a touch panel function, and accepts various operations on display data such as content, such as tapping, sliding, scrolling, etc., performed by a user's finger or stylus. An operation performed on an area of ​​the screen where the content is displayed may be regarded as an operation on the content. The terminal device 10 may be not only a smart device, but also an information processing device such as a desktop PC (Personal Computer) or a notebook PC.

[0015] Here, the terminal device 10 acquires its own position information and sensor data, and converts (vectorizes) the acquired position information and sensor data into a vector. That is, the terminal device 10 performs so-called embedding, and generates an embedding vector based on the position information and the sensor data. Then, the terminal device 10 transmits the embedding vector based on the position information and the sensor data to the information providing device 100. The sensor data is, for example, environmental sound, acceleration, air pressure, etc.

[0016] The information providing device 100 is an information processing device that receives an embedding vector based on location information and sensor data from the terminal device 10, and analyzes the behavior (location, behavior, movement, etc.) of the user U based on the embedding vector based on the location information and the sensor data, and is realized by a server device, a cloud system, etc.

[0017] [1-1. Advantages of converting location information into vectors] First, the advantages of converting position information into a vector will be described.

[0018] Conventionally, the terminal device 10 transmits location information to the information providing device 100, but at this time, the terminal device 10 performs fractional processing (rounding processing) on ​​the location information to round and convert the location information into coarse location information before transmitting it in order to consider the privacy of the user U and reduce the amount of data communication. For example, when the location information is latitude and longitude information of GPS (Global Positioning System) coordinates, the values ​​after the first decimal place in the decimal angle of the latitude and longitude are rounded up or down (rounding is also possible) to convert it into location information of a rough value.

[0019] In this case, when the information providing device 100 tries to estimate the behavior of the user U based on the position information, the rounded position information is a rough value, so that although it can identify a wide area such as a region or a large facility, it may not be possible to identify a narrow area such as an adjacent small building or room (and / or the difference between indoors and outdoors). Therefore, although it is possible to estimate the rough behavior of the user U, it may not be possible to estimate the detailed behavior of the user U. For example, if small buildings A and B (e.g., a convenience store and a fast food restaurant) are built adjacent to each other within the range of the location indicated by the rounded position information of the user U, it may not be possible to identify from the position information whether the user U is in building A or building B, or whether the user U is inside or outside the building (or site). What the information providing device 100 (server side) really wants to know is not the user U's position information itself, but the behavior of the user U indicated by the position information (i.e., the context of the user U).

[0020] This time, when the terminal device 10 converts its own location information into a vector (vectorization), it converts raw data of the location information before performing fractional processing (rounding processing) into a vector. Therefore, the terminal device 10 can generate an embedding vector after identifying a small place such as an adjacent small building or room (and / or the difference between indoors and outdoors). Therefore, the information providing device 100 can know the detailed behavior of the user U based on the embedding vector. That is, the information providing device 100 can obtain more specific and detailed information on the behavior of the user U from the terminal device 10 than by receiving conventional rounded location information. In addition, since the information providing device 100 (server side) defines the vector value of the embedding vector and its meaning, it is safe because a third party cannot understand what the vector value means even if they only see the vector value of the embedding vector.

[0021] [1-2. Overview of information processing method for embedding and transmitting location information and sensor data] Next, an overview of an information processing method for embedding and transmitting position information and sensor data will be described with reference to FIG.

[0022] As shown in FIG. 1, the terminal device 10 of a user U acquires its own location information (step S1).

[0023] Then, the terminal device 10 acquires the sensor data together with the location information (step S2).

[0024] Then, the terminal device 10 generates a pair of the location information and the sensor data (step S3).

[0025] Then, the terminal device 10 performs embedding on the pairs of location information and sensor data, converts (vectorizes) multiple pairs of location information and sensor data into one vector, and generates an embedding vector based on the location information and sensor data (step S4).

[0026] Then, the terminal device 10 transmits an embedding vector based on the position information and the sensor data to the information providing device 100 (step S5).

[0027] Then, the information providing device 100 receives an embedding vector based on the location information and the sensor data from the terminal device 10 (step S6).

[0028] Then, the information providing device 100 analyzes the behavior (location, behavior, movement, etc.) of the user U based on the embedding vector received from the terminal device 10 (step S7).

[0029] 1-3. Example 1 Next, a first embodiment of the information processing method for embedding and transmitting the above-mentioned position information will be described.

[0030] For example, in step S1, the terminal device 10 possessed by each of a plurality of users U located in each region acquires its own position information using a positioning technique such as a Global Navigation Satellite System (GNSS) or an indoor positioning technique. At this time, the terminal device 10 acquires time information indicating the time when the position information was acquired together with its own position information.

[0031] At the same time, in step S2, the terminal device 10 acquires sensor data such as environmental sound, acceleration, and air pressure, together with the location information, using sensors mounted on or connected to the terminal device 10 itself.

[0032] For example, the sensors may be detection devices other than the terminal device 10, such as wearable devices and wireless devices.

[0033] In this way, by acquiring sensor data in addition to position information, the accuracy of estimation of the user environment (context of user U, etc.) is improved. For example, sensor data such as environmental sound, acceleration, and air pressure of user U supports the position information and the user environment. In addition, changes in the user environment (changes in environmental sound, acceleration, air pressure, etc.) at the same position can be recognized.

[0034] At this time, the terminal device 10 accumulates the acquired position information as a position history (for example, a location history, an action history, etc.). The terminal device 10 also accumulates the acquired various sensor data by type. The terminal device 10 may accumulate the position information and the sensor data by linking them to each other for each acquired time or time period.

[0035] Next, in step S3, the terminal device 10 generates pairs of location information and sensor data at the same time or in the same time period, and generates sequence data (time-series data) in which the pairs of location information and sensor data are arranged in chronological order.

[0036] Next, in step S4, the terminal device 10 generates an embedding vector based on sequence data in which pairs of location information and sensor data are arranged in a time series. Here, the terminal device 10 converts the sequence data into a multidimensional real-valued vector using a machine learning technique using a neural network or the like, with the sequence data as a data set. At this time, the terminal device 10 inputs the sequence data into a vector conversion model constructed through machine learning using, for example, a recurrent neural network (RNN) or a long short-term memory (LSTM), and obtains an embedding vector as an output. The terminal device 10 may receive the vector conversion model from the information providing device 100.

[0037] The RNN and LSTM may be neural networks based on the mechanism of attention. Attention can handle data such as sentences in which the order of precedence and followance is important. The terminal device 10 may also use a similar natural language processing model. By using such a model to generate embedding vectors from sequence data, it is possible to realize generation of embedding vectors that place more emphasis on the order of information.

[0038] Alternatively, the terminal device 10 may convert the sequence data into an embedding vector in accordance with a predetermined conversion rule (rule-based).

[0039] However, the sequence data is merely an example, and in reality, for example, raw data of position information may be input directly to the model in order along the time series.

[0040] Here, when the terminal device 10 embeds based on the data of the pair of location information and sensor data according to the above vector conversion model or a predetermined conversion rule, the output value becomes a vector expression. When these output vectors are arranged in a chronological order, the user's movement route is shown. For example, although the specific latitude and longitude are unknown, it is shown that the user moved from point A to point B in the vector space. In other words, the result (output value) of the embedding by the terminal device 10 using the pair of location information and sensor data is the embedding vector. This embedding vector is a value indicating the user's behavior (a value indicating the context).

[0041] The timing at which the terminal device 10 performs embedding is arbitrary. For example, the accumulated location information may be embedded at the timing when transmission of location information from the terminal device 10 to the information providing device 100 is permitted. Alternatively, accumulation of location information may be started when an application (app) is started, and the accumulated location information may be embedded at the timing when use of the app is terminated. Alternatively, the accumulated location information may be embedded when the number of pieces of accumulated location information reaches a predetermined number (for example, 100 pieces). Alternatively, the accumulated location information may be embedded when the user moves from a place where the user had been staying (when the location information changes significantly).

[0042] Next, in step S5, the terminal device 10 transmits an embedding vector based on the pair of the position information and the sensor data to the information providing device 100 via the network N (see FIG. 3).

[0043] The timing at which the terminal device 10 transmits the embedding vector based on the pair of location information and sensor data to the information providing device 100 is arbitrary. For example, it may be a timing at which transmission of location information from the terminal device 10 to the information providing device 100 is permitted. It may also be when an application (app) is started or in use, at a predetermined time, or at a predetermined cycle (every two hours, etc.). The information providing device 100 may also request the terminal device 10 to transmit an embedding vector based on the pair of location information and sensor data.

[0044] Furthermore, when the timing is such that transmission of location information from the terminal device 10 to the information providing device 100 is permitted, the terminal device 10 may transmit its own location information at that time to the information providing device 100 together with an embedding vector based on a pair of the location information and the sensor data. That is, the terminal device 10 may transmit to the information providing device 100 a pair of its own location information and an embedding vector based on a pair of the location information and the sensor data.

[0045] Next, in step S6, the information providing device 100 receives, via the network N, embedding vectors based on pairs of location information and sensor data from the terminal devices 10 carried by each of the multiple users U located in each area.

[0046] Next, in step S7, the information providing device 100 places the embedding vector in a specific space. By placing the embedding vector in the specific space, the information providing device 100 can clearly grasp the similarity or change in the user U's own behavior, and / or the similarity or difference between the user U's behavior and that of other users. For example, the terminal device 10 performs embedding on a set of location information (latitude and longitude, etc.) and sensor data, and therefore generates embedding vectors of the same value when the terminal device 10 is in the same position. The information providing device 100 places embedding vectors of the same value at the same position in the specific space.

[0047] In this case, since the information providing device 100 is not mapped (associated) with a position on a map (latitude, longitude, etc.), it cannot specify a specific position, but it is possible to confirm specific actions of the user U, such as the user U moving from point A to point B during this time period, then moving to point C, and then returning to point A again. Furthermore, the information providing device 100 can confirm the fact of the user U's movement without using position information.

[0048] As described above, in this embodiment, when transmitting the user's location information and sensor data, the terminal device does not transmit the location information and sensor data acquired by the terminal device to the receiving side (server side) as is, but embeds the location information and sensor data before transmitting them to the receiving side. Since embedded information is difficult to restore on the receiving side, it is difficult to identify accurate location information and sensor data, and the problem of personal information is solved. In other words, not only is the amount of information reduced, but the raw data (location information and sensor data itself) detected by the terminal side cannot be restored by the server side, so it is secure. In addition, by converting a chunk of location information and sensor data into a vector indicating a specific action taken by the user (going to work, shopping, traveling by train, etc.) through embedding and transmitting it, it becomes unnecessary to transmit each piece of location information and sensor data one by one, and the problem of battery consumption is also solved.

[0049] Furthermore, in this embodiment, the amount of data can be significantly reduced by combining information such as environmental sound, acceleration, atmospheric pressure, etc. with position information, embedding it, and transmitting it. If this information were transmitted as raw data, it would become large in volume, but by embedding it, it can be transmitted as a single vector.

[0050] 1-4. Example 2 Next, a second embodiment of the information processing method for embedding and transmitting the above-mentioned position information will be described.

[0051] For example, in step S5, the terminal device 10 may transmit some of its own location information together with an embedding vector based on a pair of the location information and the sensor data to the information providing device 100. That is, the terminal device 10 may transmit to the information providing device 100 a pair of the embedding vector based on the pair of the location information and the sensor data and its own location information.

[0052] For example, when there was no restriction on the transmission of location information (when location information could be freely transmitted), the terminal side transmitted about 17 pieces of location information to the server side on average in one day in the case of "always permitted", but recently the restriction has become stricter, and the terminal side transmits only about 3 pieces of location information to the server side. Therefore, the terminal device 10 may generate an embedding vector by vectorizing 14 pieces of location information out of the 17 pieces of location information, for example, and transmit it to the server side together with the remaining 3 pieces of location information. That is, the terminal device 10 may transmit embedding information by vectorizing a part of the location information together with the sensor data, and the remaining location information itself. Note that the remaining location information is the number of pieces of location information that the terminal side can transmit. The terminal device 10 may also transmit the sensor data that is paired with the remaining location information.

[0053] The embedding vector indicates the behavior of the user U, but does not indicate the specific location of the user U. Therefore, the information providing device 100 (server side) cannot know the area in which the user U's behavior was performed. Therefore, by combining the embedding vector with the rounded location information, the information providing device 100 (server side) can know the approximate location in which the user U's behavior was performed. For example, the embedding vector can indicate that "the user U had lunch," but does not indicate "the area in which" the user U had lunch. By adding the rounded location information indicating the approximate location (for example, XX ward, △△ town), the approximate location in which the user U had lunch can be known.

[0054] This may also be implemented in combination with the above-mentioned embodiment 1. For example, in step S7, when arranging the embedding vector in a specific space, the information providing device 100 can map the position of the specific space and the position on the map (latitude, longitude, etc.) by using three points of position information to map the specific space and the map. This allows the information providing device 100 to estimate the location to a certain extent, although not completely.

[0055] Furthermore, since the terminal device 10 performs embedding on the location information (latitude, longitude, etc.), an embedding vector with the same value is generated when the user is in the same location. Therefore, the information providing device 100 can also estimate that the user U is staying in the same place based on the embedding vector with the same value. As a result, even if the information providing device 100 does not specify a specific location, it can estimate that the user U is "at home" when the user U routinely stays in the same place at night for a long time, and that the user U is "at work" when the user U routinely stays in a place different from the night for a long time during the day.

[0056] 1-5. Example 3 Next, a third embodiment of the information processing method for embedding and transmitting the above-mentioned position information will be described.

[0057] For example, in step S3, the terminal device 10 may generate an embedding vector that indicates the behavior of user U (user context) represented by two or more pairs of location information and sensor data as vector values ​​based on sequence data in which pairs of location information and sensor data are arranged in chronological order.

[0058] For example, the terminal device 10 inputs sequence data into a vector conversion model generated through machine learning using, for example, RNN or LSTM, and obtains an embedding vector as an output. At this time, even if the data is the same, the RNN or LSTM outputs differently depending on the order in which the data is input. For example, if the user moves in the order of "position 1" → "position 2" → "position 3", the output is "went to work" (outbound), and if the user moves in the order of "position 3" → "position 2" → "position 1", the output is "went home" (return). By using a model that provides information according to the time series of such information, it is possible to generate an embedding vector based on the characteristics of the transition of the position information.

[0059] In this way, when an embedding vector is generated from sequence data using LSTM, it is possible to generate an embedding vector indicating whether or not the user performed a predetermined action from the action of user U indicated by the sequence data (the context of user U). For example, when sequence data of a pair of location information and sensor data indicates that user U moved from an area where his / her home is located to an area where his / her workplace is located, the terminal device 10 can generate an embedding vector in which an element corresponding to the action of "going to work" takes a predetermined value.

[0060] The RNN and LSTM may be neural networks based on an attention mechanism. The terminal device 10 may use a similar natural language processing model.

[0061] Alternatively, the terminal device 10 may convert the sequence data into an embedding vector according to a predetermined conversion rule. For example, if the first acquired location information #1 is within a predetermined range from the user U's home, the second acquired location information #2 is within a predetermined range from the station, and the last acquired location information #3 is within a predetermined range from the workplace, the terminal device 10 may generate an embedding vector in which an element corresponding to the action of "going to work" takes a predetermined value.

[0062] However, the sequence data is merely an example, and in reality, for example, raw data of position information may be input directly to the model in order along the time series.

[0063] In addition, if there is no significant change in the location information acquired over a period of, for example, 30 minutes, the terminal device 10 may calculate the average value, median value, etc. of the location information acquired over that period, use this as the location information for that period, and generate an embedding vector from that location information.

[0064] Then, in step S7, the information providing device 100 determines the behavior (user context) of the user U, such as "at home," "went to work," "went shopping," or "traveled by train," based on the vector value of the embedding vector based on the pair of location information and sensor data. At this time, the information providing device 100 may determine the behavior for each time period based on values ​​indicated by multiple elements of the multidimensional embedding vector. For example, when the embedding vector includes a value indicating "went out in the morning" and a value indicating "went to work," the information providing device 100 may determine that the user "went to work in the morning."

[0065] Here, the values ​​of the elements of the embedding vector correspond to the behavior of the user U. For example, in the case of a three-dimensional vector, the value of the first element may indicate "whether or not the user went to work", the value of the second element may indicate "whether or not the user ate out", and the value of the third element may indicate "whether or not the user took the train". In this case, the embedding vector may be a one-hot vector. A one-hot vector is a vector such as (1,0,0) in which one component is 1 and the remaining components are all 0. However, in practice, the embedding vector is not limited to a one-hot vector.

[0066] Furthermore, the values ​​of the elements of the embedding vector may correspond to multiple actions of the user U. For example, the value of the first element may indicate "whether or not the user went to work and ate out," and the value of the second element may indicate "whether or not the user did not go to work and ate out."

[0067] The values ​​of multiple elements included in a certain range of the embedding vector may correspond to the actions of the user U. For example, the values ​​up to the third element (the upper three digits) may be stipulated such that "100" indicates "went to work" and "101" indicates "went to work by train."

[0068] The value of each element of the embedding vector may be an evaluation value. For example, it may be specified that the value (top three digits) of the first three elements is "000" to indicate "not moving", and "100" to indicate "moved 100 kilometers". The value of any element of the embedding vector may be a value indicating the moving direction. For example, the values ​​of the fourth to seventh elements are "0000" to indicate the moving direction is "north", "0001" to indicate the moving direction is "north-northeast", "0010" to indicate the moving direction is "northeast", "0011" to indicate the moving direction is "east-northeast", "0100" to indicate the moving direction is "east", "1000" to indicate the moving direction is "south", and "1100" to indicate the moving direction is "west". The value of each element of the embedding vector may be a score of a healthy life.

[0069] It is assumed that the element values ​​of the embedding vector can be arbitrarily set when learning the model or creating the conversion rule. For example, the server side can arbitrarily set the element values ​​according to the circumstances such as what kind of behavioral information the server side wants to obtain from the user U and use in the service it provides to the user U.

[0070] Here, the vector conversion model, the predetermined conversion rule, etc. for converting the pair of location information and sensor data into an embedding vector are simple vector conversion models, conversion rules, etc. for the purpose of converting sequence data of the pair of location information and sensor data into vector values ​​indicating a specific behavior of the user U, such as "at home," "went to work," "went shopping," "traveling by train," etc. For example, when the information providing device 100 wants to know whether the user U has performed a specific behavior, it provides the terminal device 10 of the user U with a vector conversion model, the predetermined conversion rule, etc. for converting the pair of location information and sensor data related to the behavior into an embedding vector.

[0071] [1-6. Overview of the method for predicting future data of a combination of location information and sensor data] Next, an overview of a method for predicting future data of a set of position information and sensor data will be described with reference to Fig. 2. Fig. 2 is an explanatory diagram showing an overview of a method for predicting future data according to an embodiment.

[0072] The terminal device 10 of the user U predicts future data based on past data of a pair of location information and sensor data as necessary. Note that the prediction of future data of a pair of location information and sensor data is merely an example. In reality, the terminal device 10 does not need to predict future data of a pair of location information and sensor data.

[0073] As shown in FIG. 2, when the terminal device 10 of the user U generates a pair of location information and sensor data (step S3 in FIG. 1), the terminal device 10 performs machine learning on the pair of location information and sensor data using a machine learning technique such as a neural network, and constructs a predictive model of future data based on past data.

[0074] In the example shown in Fig. 2, a pair of location information P and sensor data SD1 and SD2 is generated for each time T, and machine learning of distributed representation is performed as a dataset D. Here, the sensor data SD1 indicates environmental sound, and the sensor data SD2 indicates acceleration. For example, when the time T is "t-2", a pair of location information P1 and sensor data SD11 and SD21 is generated and set as dataset D1.

[0075] Here, the terminal device 10 trains the prediction model so that when a pair of location information and sensor data at each of times "t-2," "t-1," and "t" is input, a pair of location information and sensor data at each of the subsequent times "t+1," "t+2," and "t+3" is output. In this way, when sensor data is combined with location information at past and current times, sensor data corresponding to location information at a future time is obtained.

[0076] For example, the prediction model is an encoder-decoder model. This is a model that uses two RNNs, and if it is trained in the order "(past data) → first RNN (encoder) → vector → second RNN (decoder) → (future data)", the vector output by the first RNN will successfully capture the characteristics of the data. If it is unable to successfully capture the characteristics of the data, the accuracy of the future data will decrease. Conversely, if the first RNN is modified so that the accuracy of the future data is increased, the accuracy of the vector will improve.

[0077] Here, the terminal device 10 generates distributed representations using the encoder of the encoder-decoder model. Then, the terminal device 10 predicts the future from the generated distributed representations using the decoder. In this way, the terminal device 10 estimates pairs of location information and sensor data to be acquired in the future from pairs of past location information and sensor data.

[0078] In this way, the encoder and decoder are separate RNNs. When training the encoder, the encoder and decoder are modified together so that the decoder can properly predict the future from the distributed representations output by the encoder, thereby improving the performance of the encoder.

[0079] As a result, even if a situation occurs in which sensor data cannot be acquired at a certain time, a pair of location information and sensor data up to that time can be input to the prediction model, and the sensor data that should have been acquired at that time can be estimated based on the pair of location information and sensor data obtained as an output. In addition, it becomes possible to generate an embedding vector that predicts the future behavior of user U, not just an embedding vector that indicates the past or present behavior of user U. In addition, the generation accuracy of the embedding vector is improved.

[0080] [2. Example of information processing system configuration] Next, a configuration of an information processing system 1 including an information providing device 100 according to an embodiment will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of a configuration of the information processing system 1 according to an embodiment. As shown in Fig. 3, the information processing system 1 according to an embodiment includes a terminal device 10 and an information providing device 100. These various devices are connected to each other via a network N so as to be able to communicate with each other by wire or wirelessly. The network N is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network) such as the Internet.

[0081] Furthermore, the number of devices included in the information processing system 1 shown in Fig. 3 is not limited to that shown in the figure. For example, in Fig. 3, for the sake of simplicity, only one terminal device 10 is shown, but this is merely an example and is not limiting, and two or more devices may be included.

[0082] The terminal device 10 is an information processing device used by a user U. For example, the terminal device 10 is a smart device such as a smartphone or a tablet terminal, a feature phone, a PC (Personal Computer), a PDA (Personal Digital Assistant), a car navigation system, a wearable device such as a smart watch or a head-mounted display, smart glasses, or the like.

[0083] In addition, the terminal device 10 can connect to a network N via a wireless communication network such as LTE (Long Term Evolution), 4G (4th Generation), or 5G (5th Generation: 5th generation mobile communication system), or via short-range wireless communication such as Bluetooth (registered trademark) or wireless LAN (Local Area Network), and communicate with the information providing device 100.

[0084] The information providing device 100 is, for example, a PC, a server device, a mainframe, a workstation, etc. The information providing device 100 may be realized by cloud computing.

[0085] [3. Example of terminal device configuration] Next, the configuration of the terminal device 10 will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of the configuration of the terminal device 10. As shown in Fig. 4, the terminal device 10 includes a communication unit 11, a display unit 12, an input unit 13, a positioning unit 14, a sensor unit 20, a control unit 30 (controller), and a storage unit 40.

[0086] (Communications Department 11) The communication unit 11 is connected to a network N (see FIG. 3) by wired or wireless connection, and transmits and receives information to and from the information providing device 100 via the network N. For example, the communication unit 11 is realized by a NIC (Network Interface Card), an antenna, or the like.

[0087] (Display section 12) The display unit 12 is a display device that displays various information such as position information. For example, the display unit 12 is a liquid crystal display (LCD) or an organic electro-luminescent display (OLED). The display unit 12 is a touch panel display, but is not limited to this.

[0088] (Input section 13) The input unit 13 is an input device that accepts various operations from the user U. The input unit 13 has, for example, buttons for inputting characters, numbers, and the like. In addition, when the display unit 12 is a touch panel display, a part of the display unit 12 functions as the input unit 13. The input unit 13 may be a microphone that accepts voice input from the user U. The microphone may be wireless.

[0089] (Positioning unit 14) The positioning unit 14 receives a signal (radio wave) transmitted from a satellite of a GPS (Global Positioning System), and acquires position information (e.g., latitude and longitude) indicating the current position of the terminal device 10, which is the device itself, based on the received signal. That is, the positioning unit 14 measures the position of the terminal device 10. Note that the GPS is merely an example of a GNSS (Global Navigation Satellite System).

[0090] The positioning unit 14 can measure the position by various methods other than GPS. For example, the positioning unit 14 may measure the position by using various communication functions of the terminal device 10 as described below as an auxiliary positioning means for position correction or the like.

[0091] (Wi-Fi positioning) For example, the positioning unit 14 uses a Wi-Fi (registered trademark) communication function of the terminal device 10 or a communication network provided by each communication company to measure the position of the terminal device 10. Specifically, the positioning unit 14 performs Wi-Fi communication or the like and measures the distance to a nearby base station or access point to measure the position of the terminal device 10.

[0092] (Beacon positioning) The positioning unit 14 may measure the position by using a Bluetooth (registered trademark) function of the terminal device 10. For example, the positioning unit 14 measures the position of the terminal device 10 by connecting to a beacon transmitter connected by the Bluetooth (registered trademark) function.

[0093] (geomagnetic positioning) The positioning unit 14 also measures the position of the terminal device 10 based on a geomagnetic pattern of a structure measured in advance and a geomagnetic sensor included in the terminal device 10.

[0094] (RFID positioning) Furthermore, for example, if the terminal device 10 has a function of an RFID (Radio Frequency Identification) tag equivalent to a contactless IC card used at station ticket gates, stores, etc., or has a function of reading an RFID tag, the location of use is recorded together with information on a payment or the like made by the terminal device 10. The positioning unit 14 may obtain such information to measure the location of the terminal device 10. Furthermore, the location may be measured by an optical sensor, an infrared sensor, or the like provided in the terminal device 10.

[0095] The positioning unit 14 may, if necessary, measure the position of the terminal device 10 using one or a combination of the above-mentioned positioning means.

[0096] (Sensor unit 20) The sensor unit 20 includes various sensors mounted on or connected to the terminal device 10. The connection may be wired or wireless. For example, the sensor unit 20 may be a detection device other than the terminal device 10, such as a wearable device or a wireless device. In the example shown in FIG. 4, the sensor unit 20 includes an acceleration sensor 21, a gyro sensor 22, an air pressure sensor 23, a temperature sensor 24, a sound sensor 25, a light sensor 26, a magnetic sensor 27, and an image sensor (camera) 28.

[0097] The above-mentioned sensors 21 to 28 are merely examples and are not intended to be limiting. That is, the sensor unit 20 may be configured to include some of the sensors 21 to 28, or may include other sensors such as a humidity sensor in addition to or instead of the sensors 21 to 28.

[0098] The acceleration sensor 21 is, for example, a three-axis acceleration sensor, and detects physical movements of the terminal device 10, such as the moving direction, speed, and acceleration of the terminal device 10. The gyro sensor 22 detects physical movements of the terminal device 10, such as tilts in three axial directions, based on the angular velocity, etc. of the terminal device 10. The air pressure sensor 23 detects, for example, the air pressure around the terminal device 10.

[0099] Since the terminal device 10 includes the above-mentioned acceleration sensor 21, gyro sensor 22, and air pressure sensor 23, it is possible to measure the position of the terminal device 10 using a technique such as Pedestrian Dead-Reckoning (PDR) that uses these sensors 21 to 23. This makes it possible to obtain indoor position information that is difficult to obtain using a positioning system such as GPS.

[0100] For example, the number of steps, walking speed, and distance walked can be calculated by a pedometer using the acceleration sensor 21. In addition, the moving direction, line of sight direction, and body inclination of the user U can be known by using the gyro sensor 22. In addition, the altitude and floor number on which the terminal device 10 of the user U is located can be known from the air pressure detected by the air pressure sensor 23.

[0101] The air temperature sensor 24 detects, for example, the air temperature around the terminal device 10. The sound sensor 25 detects, for example, sounds (environmental sounds, etc.) around the terminal device 10. The light sensor 26 detects the illuminance around the terminal device 10. The magnetic sensor 27 detects, for example, the geomagnetism around the terminal device 10. The image sensor 28 captures an image around the terminal device 10.

[0102] The above-mentioned air pressure sensor 23, temperature sensor 24, sound sensor 25, light sensor 26, and image sensor 28 can detect the air pressure, temperature, sound, and illuminance, and capture images of the surroundings, thereby detecting the environment and situation around the terminal device 10. In addition, the accuracy of the location information of the terminal device 10 can be improved based on the environment and situation around the terminal device 10.

[0103] (Control unit 30) The control unit 30 includes, for example, a microcomputer having a central processing unit (CPU), a read only memory (ROM), a RAM, an input / output port, and various other circuits. The control unit 30 may also be configured with hardware such as an integrated circuit, for example, an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). The control unit 30 includes a transmission unit 31, a reception unit 32, a processing unit 33, an acquisition unit 34, a conversion unit 35, a prediction unit 36, and a learning unit 37.

[0104] (Transmitter 31) The transmission unit 31 can transmit, for example, various pieces of information input by the user U using the input unit 13, various pieces of information processed by the processing unit 33, etc. to the information providing device 100 via the communication unit 11.

[0105] Furthermore, the transmission unit 31 transmits the embedding vector converted from the position information by the conversion unit 35 to the information providing device 100 via the communication unit 11. At this time, the transmission unit 31 may transmit the position information at that time together with the embedding vector.

[0106] (Receiving unit 32) The receiving unit 32 receives various information provided from the information providing device 100 via the communication unit 11. For example, the receiving unit 32 receives a vector conversion model or the like provided from the information providing device 100 via the communication unit 11.

[0107] (Processing section 33) The processing unit 33 controls the entire terminal device 10, including the display unit 12. For example, the processing unit 33 can output various information transmitted by the transmission unit 31 and various information received by the reception unit 32 from the information providing device 100 to the display unit 12 for display.

[0108] (Acquisition part 34) The acquisition unit 34 acquires the position information of the terminal device 10 constantly / periodically / at a predetermined timing. For example, the acquisition unit 34 acquires the position information of the terminal device 10 whose position is measured by the positioning unit 14. The acquisition unit 34 also acquires sensor data detected by each of the sensors 21-28. The acquisition unit 34 also estimates and acquires indoor position information of the terminal device 10 based on the sensor data detected by each of the sensors 21-28.

[0109] The acquisition unit 34 also stores (saves) the acquired position information and sensor data in the storage unit 40. For example, the acquisition unit 34 accumulates the acquired position information as a position history (for example, a location history, an action history, etc.). The acquisition unit 34 also accumulates the sensor data detected by each of the sensors 21 to 28 by type. Note that the acquisition unit 34 may accumulate the position information and the sensor data in association with each other for each acquired time or time period.

[0110] In addition, the acquisition unit 34 acquires the vector conversion model and the prediction model received by the reception unit 32 from the information providing device 100, and stores (preserves) them in the storage unit 40.

[0111] Alternatively, the acquisition unit 34 acquires a predetermined conversion rule received by the reception unit 32 from the information providing device 100, and stores (preserves) the rule in the storage unit 40.

[0112] (Conversion section 35) The conversion unit 35 performs embedding on pairs of position information and sensor data, converts (vectorizes) multiple pairs of position information and sensor data into one vector, and generates an embedding vector based on the position information and the sensor data.

[0113] For example, the conversion unit 35 generates a pair of location information and sensor data at the same time or in the same time period, and generates sequence data in which the pair of location information and sensor data is arranged in chronological order. Then, the conversion unit 35 generates an embedding vector based on the sequence data.

[0114] For example, the conversion unit 35 converts the sequence data into a multidimensional real-valued vector using a machine learning technique based on a neural network, etc., with the sequence data as a data set. That is, the conversion unit 35 inputs the sequence data into a vector conversion model constructed through machine learning using, for example, a recurrent neural network (RNN) or a long short-term memory (LSTM), etc., and obtains an embedding vector as an output.

[0115] The RNN and LSTM may be neural networks based on an attention mechanism. The conversion unit 35 may use a similar natural language processing model.

[0116] Alternatively, the conversion unit 35 may convert the sequence data into an embedding vector in accordance with a predetermined conversion rule (rule-based).

[0117] However, the sequence data is merely an example, and in reality, for example, raw data such as position information and sensor data may be input directly to the model in order along the time series.

[0118] (Prediction Section 36) The prediction unit 36 ​​predicts future data based on past data of pairs of location information and sensor data. For example, even if a situation occurs in which sensor data cannot be acquired at a certain time, the prediction unit 36 ​​inputs pairs of location information and sensor data up to that time into a prediction model, and estimates the sensor data that should have been acquired at that time based on the pairs of location information and sensor data obtained as an output.

[0119] (Learning Section 37) The learning unit 37 performs machine learning on the pair of location information and sensor data using a machine learning technique using a neural network or the like, and constructs a vector conversion model for converting the pair of location information and sensor data into an embedding vector.

[0120] Furthermore, the learning unit 37 performs machine learning on pairs of location information and sensor data using a machine learning technique using a neural network or the like, and constructs a prediction model for future data based on past data.

[0121] For example, the learning unit 37 constructs a vector conversion model or a prediction model by machine learning using a recurrent neural network (RNN) or a long short-term memory (LSTM). Note that the model may be of any type and any format. For example, the learning unit 37 may employ a support vector machine (SVM) or a deep neural network (DNN) as a model. Here, the DNN may be a convolutional neural network (CNN) or an RNN. The model may be realized by combining a plurality of models, such as a model combining a CNN and an RNN.

[0122] The RNN and LSTM may be neural networks based on an attention mechanism. The learning unit 37 may use a similar natural language processing model.

[0123] The learning is, for example, deep learning using DNN. Data mining or other machine learning algorithms may also be used. The learning unit 37 learns the model using the various learning methods described above.

[0124] (Storage unit 40) The storage unit 40 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk drive (HDD), a solid state drive (SSD), or an optical disk. Various programs and various data are stored in the storage unit 40. Here, the storage unit 40 stores, for example, various information input by the user U using the input unit 13, sensor data detected by the sensors 21 to 28 mounted on or connected to the terminal device 10, and position information of the terminal device 10 measured by the positioning unit 14. In addition, the storage unit 40 stores various information processed by the processing unit 33 and various information received by the receiving unit 32. In addition, the storage unit 40 stores a vector conversion model, a prediction model, a predetermined conversion rule, and the like.

[0125] 4. Configuration example of information processing device Next, a configuration of the information providing device 100 according to the embodiment will be described with reference to Fig. 5. Fig. 5 is a diagram showing an example of a configuration of the information providing device 100 according to the embodiment. As shown in Fig. 5, the information providing device 100 has a communication unit 110, a storage unit 120, and a control unit 130.

[0126] (Communication unit 110) The communication unit 110 is realized by, for example, a network interface card (NIC), etc. The communication unit 110 is also connected to a network N (see FIG. 3) in a wired or wireless manner.

[0127] (Storage unit 120) The storage unit 120 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. As shown in FIG. 5, the storage unit 120 has a conversion information database 121 and a vector information database 122.

[0128] (Conversion Information Database 121) The conversion information database 121 stores various information related to vector conversion models, prediction models, or predetermined conversion rules, etc. Fig. 6 is a diagram showing an example of the conversion information database 121. In the example shown in Fig. 6, the conversion information database 121 has items such as "conversion ID", "conversion information", and "recipient".

[0129] The "conversion ID" indicates identification information for identifying a vector conversion model, a prediction model, or a predetermined conversion rule, etc. The common model indicates a model managed on the server side and common to each user.

[0130] Moreover, the "conversion information" indicates a vector conversion model or a prediction model identified by a conversion ID, or a file in which predetermined conversion rules, etc., have been digitized.

[0131] Furthermore, the "recipient" indicates the terminal device 10 to which the vector conversion model or prediction model identified by the conversion ID, or the predetermined conversion rule, etc. is provided. For example, the "recipient" may store a terminal ID. There may be multiple terminal devices 10 to which the data is provided. In other words, the "recipient" may store multiple terminal IDs. Note that if there is no need to specify the terminal device 10 to which the data is provided, the "recipient" item may not be included.

[0132] For example, in the example shown in Figure 6, the vector conversion model, prediction model, or specified conversion rule, which is the conversion information "conversion information #1" identified by the conversion ID "T1", is provided to the destination "destination #1".

[0133] The conversion information database 121 may store various kinds of information according to the purpose, not limited to the above. For example, the conversion information database 121 may store detailed information of a common model identified by a conversion ID.

[0134] (Vector Information Database 122) The vector information database 122 stores various information related to embedding vectors based on a pair of position information and sensor data received from the terminal device 10. Fig. 7 is a diagram showing an example of the vector information database 122. In the example shown in Fig. 7, the vector information database 122 has items such as "Terminal ID", "Embedding Vector", and "Determination Result".

[0135] "Terminal ID" indicates identification information for identifying the terminal device 10 that is the source of the embedding vector. The "Terminal ID" may be any number assigned by the server side, or may be contact information (telephone number, email address, etc.) of the terminal device 10. If there is no need to identify the terminal device 10, the "Terminal ID" item may be omitted.

[0136] Moreover, the "embedding vector" indicates an embedding vector based on a pair of location information and sensor data received from the terminal device 10. Note that there may be multiple "embedding vectors" for one terminal device 10. For example, if the terminal device 10 generates an embedding vector every 30 minutes, the embedding vector for every 30 minutes is stored.

[0137] Furthermore, the "determination result" indicates the behavior (location, behavior, movement, etc.) of the user U determined based on an embedding vector based on a pair of position information and sensor data.

[0138] For example, in the example shown in Figure 7, the embedding vector "vector #11" received from terminal device 10 identified by terminal ID "terminal #1" indicates that the behavior of user U of terminal device 10 is the behavior indicated by the judgment result "judgment result #11".

[0139] The vector information database 122 may store various information according to the purpose, without being limited to the above. For example, the vector information database 122 may store identification information (vector ID, etc.) for identifying each embedding vector for each received data unit. The vector information database 122 may also store information indicating the date and time when each embedding vector was received. The vector information database 122 may also store information regarding rules for interpreting the embedding vector. The vector information database 122 may also store conversion information such as a vector conversion model, a prediction model, or a predetermined conversion rule.

[0140] (Control unit 130) Returning to Fig. 5, the description will be continued. The control unit 130 is a controller, and is realized by, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or the like, executing various programs (corresponding to an example of an information processing program) stored in a storage device inside the information providing device 100 using a storage area such as a RAM as a working area. In the example shown in Fig. 5, the control unit 130 has a collection unit 131, a determination unit 132, a generation unit 133, and a provision unit 134.

[0141] (Collection Department 131) The collection unit 131 collects embedding vectors based on pairs of position information and sensor data from each terminal device 10. That is, the collection unit 131 receives embedding vectors based on pairs of position information and sensor data from the terminal devices 10 via the communication unit 110.

[0142] (Judgment unit 132) The determination unit 132 analyzes an embedding vector based on a set of collected position information and sensor data, and determines the behavior of the user U (location, behavior, movement, etc.).

[0143] Furthermore, the determination unit 132 places the embedding vector in a specific space, and determines the similarity or change in the behavior of the user U himself / herself, and / or the similarity or difference between the behavior of the user U and that of other users.

[0144] (Generation unit 133) The generation unit 133 generates a vector conversion model, a prediction model, or a predetermined conversion rule for converting the location information and the sensor data into an embedding vector based on an existing location history (or assumed location information) that accumulates previously collected location information and an existing sensor history that accumulates sensor data.

[0145] For example, the generation unit 133 generates a vector conversion model that outputs an embedding vector when position information is input, by having a model learn a set of position information, sensor data, and an embedding vector as correct answer data.

[0146] Furthermore, the generation unit 133 performs machine learning on a set of location information and sensor data using a machine learning technique using a neural network or the like, and generates a prediction model of future data based on past data.

[0147] Here, the generating unit 133 generates a vector conversion model by machine learning using a recurrent neural network (RNN) or a long short-term memory (LSTM). Note that the model can be of any type and any format. For example, the generating unit 133 may use a support vector machine (SVM) or a deep neural network (DNN) as a model. Here, the DNN may be a convolutional neural network (CNN) or an RNN. The model may be realized by combining a plurality of models, such as a model combining a CNN and an RNN.

[0148] The RNN and LSTM may be neural networks based on an attention mechanism. The generation unit 133 may use a similar natural language processing model.

[0149] The learning is, for example, deep learning using DNN. Data mining or other machine learning algorithms may also be used. The generation unit 133 learns the model using the various learning methods described above.

[0150] In addition, the generation unit 133 may share the vector conversion model and the prediction model with the learning unit 37 of the terminal device 10.

[0151] (Provider 134) The providing unit 134 provides the generated vector transformation model, the predetermined transformation rule, and the like to the terminal device 10 of the user U. That is, the providing unit 134 transmits the generated vector transformation model, the predetermined transformation rule, and the like to the terminal device 10 of the user U via the communication unit 110.

[0152] 5. Processing Procedure Next, a processing procedure by the terminal device 10 and the information providing device 100 according to the embodiment will be described with reference to Fig. 8. Fig. 8 is a sequence diagram showing the processing procedure according to the embodiment. Note that the processing procedure shown below is repeatedly executed by the control unit 30 of the terminal device 10 or the control unit 130 of the information providing device 100.

[0153] 8, the generation unit 133 of the information providing device 100 generates a vector conversion model and a predetermined conversion rule for converting location information into an embedding vector based on an existing location history (or assumed location information) that accumulates location information of the terminal device 10 collected in the past and sensor data (step S101). For example, the generation unit 133 generates a vector conversion model that outputs an embedding vector when location information is input by having the model learn a pair of location information and an embedding vector as correct answer data.

[0154] Then, the providing unit 134 of the information providing device 100 provides the generated vector conversion model, the predetermined conversion rule, and the like to the terminal device 10 of the user U (step S102). That is, the providing unit 134 transmits the generated vector conversion model, the predetermined conversion rule, and the like to the terminal device 10 of the user U via the communication unit 110.

[0155] Then, the acquisition unit 34 of the terminal device 10 acquires the vector transformation model, the predetermined transformation rule, and the like provided by the information providing device 100 (step S103).

[0156] Then, the acquisition unit 34 of the terminal device 10 acquires the location information of the terminal device 10 constantly / periodically / at a predetermined timing (step S104).

[0157] For example, the acquisition unit 34 acquires the position information of the terminal device 10 whose position is measured by the positioning unit 14. In addition, the acquisition unit 34 estimates and acquires indoor position information of the terminal device 10 based on sensor data detected by the sensors 21-28.

[0158] Moreover, the acquisition unit 34 acquires, together with the position information, sensor data such as environmental sound, acceleration, and air pressure, using the sensors 21 to 28 mounted on or connected to the acquisition unit 34 (step S105).

[0159] The acquisition unit 34 accumulates the acquired position information as a position history (e.g., a location history, an action history, etc.). The acquisition unit 34 also accumulates the acquired various sensor data by type. The acquisition unit 34 may accumulate the position information and the sensor data by linking them to each other for each acquired time or time period.

[0160] Then, the conversion unit 35 of the terminal device 10 generates a pair of location information and sensor data at the same time or in the same time period (step S106).

[0161] Then, the conversion unit 35 generates sequence data in which pairs of the position information and the sensor data are arranged in time series (step S107).

[0162] At this time, the prediction unit 36 ​​of the terminal device 10 may predict future data based on past data of pairs of location information and sensor data. For example, even if a situation occurs in which sensor data cannot be acquired at a certain time, the prediction unit 36 ​​inputs pairs of location information and sensor data up to that time into a prediction model, and estimates the sensor data that should have been acquired at that time based on the pairs of location information and sensor data obtained as an output. In this case, the conversion unit 35 generates sequence data in which pairs of location information and sensor data of predicted future data are arranged in chronological order.

[0163] At this time, the learning unit 37 of the terminal device 10 may perform machine learning on the pair of the location information and the sensor data using a machine learning technique using a neural network or the like, and reconstruct a prediction model of future data based on past data.

[0164] Then, the conversion unit 35 generates an embedding vector based on the sequence data (step S108).

[0165] For example, the conversion unit 35 inputs sequence data to a vector conversion model constructed through machine learning using, for example, a recurrent neural network (RNN) or a long short-term memory (LSTM), and acquires an embedding vector as an output. Note that the RNN or LSTM may be a neural network based on an attention mechanism. The conversion unit 35 may also use a similar natural language processing model.

[0166] At this time, the learning unit 37 of the terminal device 10 may perform machine learning on the pair of location information and sensor data using a machine learning technique using a neural network, and reconstruct a vector conversion model for converting the pair of location information and sensor data into an embedding vector.

[0167] Alternatively, the conversion unit 35 may convert the sequence data into an embedding vector in accordance with a predetermined conversion rule (rule-based).

[0168] However, the sequence data is merely an example, and in reality, for example, raw data such as position information and sensor data may be input directly to the model in order along the time series.

[0169] Then, the transmitting unit 31 of the terminal device 10 transmits the embedding vector converted by the converting unit 35 to the information providing device 100 via the communication unit 11 (step S109).

[0170] Then, the collection unit 131 of the information providing device 100 collects embedding vectors based on pairs of position information and sensor data from each terminal device 10 (step S110). That is, the collection unit 131 receives embedding vectors based on pairs of position information and sensor data from the terminal devices 10 via the communication unit 110.

[0171] Then, the determination unit 132 of the information providing device 100 analyzes an embedding vector based on a set of the collected position information and sensor data, and determines the behavior (location, behavior, movement, etc.) of the user U (step S111).

[0172] At this time, the determination unit 132 may place the embedding vector in a specific space and determine the similarity or change in the behavior of the user U himself / herself, and / or the similarity or difference between the behavior of the user U and other users.

[0173] Furthermore, the generation unit 133 of the information providing device 100 may update the vector conversion model, the predetermined conversion rule, and the like, based on an embedding vector based on a pair of the position information and the sensor data received from the terminal device 10.

[0174] 6. Modifications The above-described terminal device 10 and information providing device 100 may be implemented in various different forms other than the above-described embodiment. Therefore, modifications of the embodiment will be described below.

[0175] In the above embodiment, the terminal device 10 generates an embedding vector based on a pair of position information and sensor data, but at this time, an embedding vector based on position information and an embedding vector based on sensor data may be generated separately and then combined to generate one embedding vector. For example, the terminal device 10 may arrange an embedding vector based on position information in the first to tenth elements of a multidimensional embedding vector having 20 or more elements, and arrange an embedding vector based on sensor data in the eleventh to twentieth elements. In this case, the embedding vector based on position information indicates "user's behavior or location" and the like, and the embedding vector based on sensor data indicates "user's environment or situation" and the like.

[0176] In this case, two types of vector conversion models may be constructed: a model that converts position information into vectors, and a model that converts sensor data into vectors. Alternatively, one vector conversion model may convert the input position information and sensor data into vectors individually, and combine these two types of vectors.

[0177] In the above embodiment, the information providing device 100 may determine the user U's actions such as "went to the mountains", "went to the sea", "went to the countryside", "went abroad", etc., based on the vector value of the embedding vector based on the pair of location information and sensor data. The information providing device 100 may also determine the user U's actions such as "went on a trip", "went on a business trip", etc. The information providing device 100 may also determine the user U's actions such as "went on a day trip", "went overnight", etc.

[0178] In the above embodiment, when the terminal device 10 does not want to inform the information providing device 100 (server side) of detailed information on the behavior of the user U, the terminal device 10 may perform fractional processing (rounding processing) of the location information when storing the location information or prior to embedding, and may round the location information to convert it into location information with low accuracy. For example, when the location information is latitude and longitude information of GPS (Global Positioning System) coordinates, the terminal device 10 rounds up or down (may also round off) the decimal points (minutes or less in the degree-minute-second method) of the latitude and longitude in decimal angles to convert it into location information with a rough value. This makes it possible to limit the content of information transmitted from the terminal device 10 to the information providing device 100 (server side) regarding information on the behavior of the user U. In addition, it is possible to reduce the amount of information of the location information as learning data for machine learning, and reduce the processing load.

[0179] In the above embodiment, the value of each element of the embedding vector is expressed as "0" and "1" for convenience, but in reality, it may be a number greater than or equal to "2", a number with multiple digits, or a number with an integer part and a decimal part. In other words, it is sufficient if it is a real number. This makes it possible to express multiple pieces of information or detailed information in each element of a multidimensional embedding vector.

[0180] In the above embodiment, the terminal device 10 accumulates the acquired location information and sensor data, but they may be accumulated in a vectorized state. For example, the terminal device 10 may convert a set of location information and sensor data into a vector value indicating "at home," "at work," "at a store," "outdoors," etc., based on the location indicated by each piece of location information, and accumulate the converted vector value. When the same vector value continues along a time series, they may be collectively treated as an embedding vector.

[0181] Furthermore, in the above embodiment, the terminal device 10 or the information providing device 100 may individually adjust the vector conversion model, the prediction model, or the predetermined conversion rule, etc., according to the characteristics and circumstances of each user U. For example, the vector conversion model, the prediction model, or the predetermined conversion rule, etc. may be customized.

[0182] In the above embodiment, a vector conversion model for converting a pair of location information and sensor data into an embedding vector, a prediction model of future data based on past data, etc. may be generated using federated learning or the like. For example, in federated learning, each terminal device 10 performs machine learning on a vector conversion model, a prediction model, etc., using a pair of location information and sensor data as learning data, and transmits difference data between the original model and the model after learning to the information providing device 100. The information providing device 100 receives the difference data from each terminal device 10, integrates the difference data, learns, and generates a common model for each model. Then, the information providing device 100 provides parameters of the generated common model to each terminal device 10. In this way, by using federated learning, it is possible to perform learning without providing a pair of location information and sensor data of the terminal, which is learning data, to the server side, so that it is possible to take into consideration the personal information and privacy of the user.

[0183] 7. Effects As described above, the information processing device (terminal device 10) according to the present application includes a first acquisition unit (acquisition unit 34) that acquires location information, a second acquisition unit (acquisition unit 34) that acquires sensor data from a sensor, a conversion unit 35 that converts the location information and the sensor data into vectors, and a transmission unit that transmits the vectors. At this time, the conversion unit 35 converts multiple pieces of location information and sensor data into a single vector all at once. The conversion unit 35 converts the location information and the sensor data into vectors using a vector conversion model obtained by machine learning or in accordance with a predetermined conversion rule.

[0184] This enables the information processing device to transmit the position information and the sensor data after embedding them.

[0185] [8. Hardware Configuration] Moreover, the terminal device 10 and the information providing device 100 according to the above-described embodiment are realized by a computer 1000 having a configuration as shown in Fig. 9, for example. The information providing device 100 will be described below as an example. Fig. 9 is a diagram showing an example of a hardware configuration. The computer 1000 is connected to an output device 1010 and an input device 1020, and has a configuration in which a calculation device 1030, a primary storage device 1040, a secondary storage device 1050, an output I / F (Interface) 1060, an input I / F 1070, and a network I / F 1080 are connected by a bus 1090.

[0186] The arithmetic device 1030 operates based on programs stored in the primary storage device 1040 and the secondary storage device 1050, programs read from the input device 1020, and the like, and executes various processes. The arithmetic device 1030 is realized by, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or the like.

[0187] The primary storage device 1040 is a memory device such as a RAM (Random Access Memory) that primarily stores data used by the arithmetic device 1030 for various calculations. The secondary storage device 1050 is a storage device in which data used by the arithmetic device 1030 for various calculations and various databases are registered, and is realized by a ROM (Read Only Memory), an HDD (Hard Disk Drive), an SSD (Solid State Drive), a flash memory, or the like. The secondary storage device 1050 may be an internal storage device or an external storage device. The secondary storage device 1050 may be a removable storage medium such as a USB memory or a Secure Digital (SD) memory card. The secondary storage device 1050 may be a cloud storage device (online storage device), a NAS (Network Attached Storage), a file server, or the like.

[0188] The output I / F 1060 is an interface for transmitting information to be output to an output device 1010 that outputs various types of information, such as a display, a projector, a printer, etc., and is realized by a connector conforming to a standard such as USB (Universal Serial Bus), DVI (Digital Visual Interface), or HDMI (registered trademark) (High Definition Multimedia Interface). The input I / F 1070 is an interface for receiving information from various input devices 1020, such as a mouse, a keyboard, a keypad, a button, a scanner, etc., and is realized by a USB, etc.

[0189] Furthermore, the output I / F 1060 and the input I / F 1070 may be wirelessly connected to the output device 1010 and the input device 1020, respectively. That is, the output device 1010 and the input device 1020 may be wireless devices.

[0190] The output device 1010 and the input device 1020 may be integrated into one device, such as a touch panel. In this case, the output I / F 1060 and the input I / F 1070 may also be integrated into one device, such as an input / output I / F.

[0191] The input device 1020 may be a device that reads information from, for example, an optical recording medium such as a CD (Compact Disc), a DVD (Digital Versatile Disc), or a PD (Phase change rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0192] The network I / F 1080 receives data from other devices via the network N and sends it to the arithmetic device 1030, and also transmits data generated by the arithmetic device 1030 to other devices via the network N.

[0193] The arithmetic unit 1030 controls the output device 1010 and the input device 1020 via the output I / F 1060 and the input I / F 1070. For example, the arithmetic unit 1030 loads a program from the input device 1020 or the secondary storage device 1050 onto the primary storage device 1040 and executes the loaded program.

[0194] For example, when the computer 1000 functions as the information providing device 100, the arithmetic unit 1030 of the computer 1000 executes a program loaded onto the primary storage device 1040 to realize the functions of the control unit 130. The arithmetic unit 1030 of the computer 1000 may also load a program acquired from another device via the network I / F 1080 onto the primary storage device 1040 and execute the loaded program. The arithmetic unit 1030 of the computer 1000 may also cooperate with the other device via the network I / F 1080 and use functions, data, etc. of a program by calling them from another program of the other device.

[0195] [9. Others] Although the embodiments of the present application have been described above, the present invention is not limited to the contents of these embodiments. The above-described components include those that can be easily imagined by a person skilled in the art, those that are substantially the same, and those that are within the so-called equivalent range. Furthermore, the above-described components can be appropriately combined. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the spirit of the above-described embodiments.

[0196] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

[0197] In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc.

[0198] For example, the information providing device 100 described above may be realized by multiple server computers, and depending on the functions, the configuration can be flexibly changed, such as by calling an external platform using an API (Application Programming Interface) or network computing.

[0199] Furthermore, the above-described embodiments and modifications can be appropriately combined as long as the processing contents are not contradictory.

[0200] Furthermore, the above-mentioned "section, module, unit" can be read as "means" or "circuit", etc. For example, an acquisition section can be read as an acquisition means or an acquisition circuit. [Explanation of symbols]

[0201] 1. Information Processing Systems 10 Terminal Equipment 14 Positioning unit 30 Control section 31 Transmitter 32 Receiving section 33 Processing section 34 Acquisition Department 35 Conversion unit 36 Prediction Department 37 Learning Department 100 Information provision device 110 Communications Department 120 Storage section 121 Conversion Information Database 122 Vector Information Database 130 Control section 131 Collection Department 132 Judgment section 133 Generation part 134 Provision Department

Claims

1. An information processing device used as a user's terminal device, A first acquisition unit that acquires raw data of location information of a user's terminal device before performing rounding; A second acquisition unit that acquires sensor data related to a situation of the user and a surrounding environment from a sensor; a conversion unit that generates a pair of raw data of the position information and the sensor data, embeds the pair of the raw data of the position information and the sensor data into a vector space related to a user's behavior, and converts the pair into an embedding vector that indicates the user's behavior based on the raw data of the position information and the sensor data; A transmitter for transmitting the embedding vector; An information processing device comprising:

2. The sensor data includes at least one of environmental sound, acceleration, and atmospheric pressure.

2. The information processing apparatus according to claim 1,

3. The conversion unit converts the plurality of pieces of position information and the plurality of pieces of sensor data into a single embedding vector.

3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

4. The conversion unit converts sequence data in which the plurality of pieces of position information and the sensor data are arranged in a time series into a multidimensional embedding vector.

4. The information processing device according to claim 1, wherein the information processing device is a computer.

5. The conversion unit converts a set of the position information and the sensor data into the embedding vector using a vector conversion model obtained by machine learning.

5. The information processing apparatus according to claim 1, wherein the information processing apparatus further comprises:

6. The conversion unit converts a set of the position information and the sensor data into the embedding vector according to a predetermined conversion rule.

6. The information processing apparatus according to claim 1, wherein the information processing apparatus further comprises:

7. a prediction unit that predicts future position information and sensor data based on the position information and the sensor data; 7. The information processing apparatus according to claim 1, further comprising:

8. The information processing device according to claim 7 , wherein the prediction unit predicts the future position information and the sensor data by using a prediction model obtained by machine learning.

9. The conversion unit converts a pair of the part of the position information and the sensor data among the plurality of pieces of position information into an embedding vector; The transmission unit transmits an embedding vector converted from a part of the position information and the sensor data, and the remaining position information.

9. The information processing apparatus according to claim 1, wherein the information processing apparatus further comprises:

10. The values ​​of the elements of the embedding vector correspond to the user's actions.

10. The information processing device according to claim 1, wherein the information processing device is a computer.

11. A collection unit that collects embedding vectors indicating the behavior of a user from a terminal device of the user based on raw data of location information of the terminal device before rounding and sensor data regarding the user's situation and surrounding environment; A determination unit that determines a behavior of a user of the terminal device based on the embedding vector; Equipped with The values ​​of the elements of the embedding vector correspond to the behavior of the user of the terminal device; and The determination unit determines a behavior of a user of the terminal device by referring to values ​​of elements of the embedding vector.

23. An information processing apparatus comprising:

12. An information processing method executed by an information processing device used as a user's terminal device, comprising: A first acquisition step of acquiring raw data of location information of a user's terminal device before performing rounding; A second acquisition step of acquiring sensor data related to the user's situation and a surrounding environment from a sensor; a conversion process of generating a pair of raw data of the position information and the sensor data, embedding the pair of the raw data of the position information and the sensor data into a vector space related to a user's behavior, and converting the pair into an embedding vector indicating the user's behavior based on the raw data of the position information and the sensor data; a transmitting step of transmitting the embedding vector; 13. An information processing method comprising:

13. A first acquisition step of acquiring raw data of location information of a user's terminal device before performing rounding processing; A second acquisition step of acquiring sensor data related to the user's situation and a surrounding environment from a sensor; a conversion procedure of generating a pair of raw data of the position information and the sensor data, embedding the pair of the raw data of the position information and the sensor data into a vector space related to a user's behavior, and converting the pair into an embedding vector indicating the user's behavior based on the raw data of the position information and the sensor data; a transmitting step of transmitting the embedding vector; An information processing program for causing a computer used as a user's terminal device to execute the above.

Images